I use curl to test out my HTTP libraries all the time. Recently, I ran into an issue where when uploading a file (25mb) from curl in the command line to my common lisp app server, only about half the data showed up (12.5mb). I was doing this:
curl -H 'Authorization: ...' -H 'Transfer-Encoding: chunked' --data-binary @/media/large_vid.mov
Naturally, I assumed the issue was with my libraries. It could be the cl-async library dropping packets, it could be the HTTP parser having issues, and it could be the app server itself. I mean, it has to be one of those. Curl has been around for ages, and there's no way it would just drop data. So I spent days tearing my hair out.
Finally, I ran curl with the --trace
and looked at the data. It provides a hex dump of everything it sends. It's not formatted perfectly, but with vim's block select and a few handy macros, I was able to get the length of the data being sent: 12.5mb. That's right, curl was defying me. There was no error in my code at all.
I did a search online for curl not sending the full file data when using --data-binary
. Nothing. So I looked over my options and found -T
which looks surprisingly similar to --data-binary
with the @
modifier. I tried:
curl -H 'Authorization: ...' -H 'Transfer-Encoding: chunked' -T /media/large_vid.mov
All 25mb came through (every byte accounted for).
Conclusion
If you're uploading files, use -T /path/to/file
instead of --data-binary @/path/to/file
. Note that -d/-D
were also "broken."