201310.01

Curl CLI not sending full file data when using --data-binary

I use curl to test out my HTTP libraries all the time. Recently, I ran into an issue where when uploading a file (25mb) from curl in the command line to my common lisp app server, only about half the data showed up (12.5mb). I was doing this:

curl -H 'Authorization: ...' -H 'Transfer-Encoding: chunked' --data-binary @/media/large_vid.mov

Naturally, I assumed the issue was with my libraries. It could be the cl-async library dropping packets, it could be the HTTP parser having issues, and it could be the app server itself. I mean, it has to be one of those. Curl has been around for ages, and there's no way it would just drop data. So I spent days tearing my hair out.

Finally, I ran curl with the --trace and looked at the data. It provides a hex dump of everything it sends. It's not formatted perfectly, but with vim's block select and a few handy macros, I was able to get the length of the data being sent: 12.5mb. That's right, curl was defying me. There was no error in my code at all.

I did a search online for curl not sending the full file data when using --data-binary. Nothing. So I looked over my options and found -T which looks surprisingly similar to --data-binary with the @ modifier. I tried:

curl -H 'Authorization: ...' -H 'Transfer-Encoding: chunked' -T /media/large_vid.mov

All 25mb came through (every byte accounted for).

Conclusion

If you're uploading files, use -T /path/to/file instead of --data-binary @/path/to/file. Note that -d/-D were also "broken."