Updates to our Python client: faster, concurrent uploads and much more.
Our Python SDK client, which lets users interact with our platform through their Python environment, just got some exciting improvements! The headline is that dataset uploads through our client are now executed concurrently by default, which lets datasets land on the platform several times faster. Alongside that, over the past few months we have been working continuously to support new features and make improvements to deliver a great developer experience. Our Python SDK can now download dataset samples, delete dataset submissions, list and search our dataset catalogue, extract dataset archives automatically, and report errors more accurately.
In order to take advantage of the latest features, simply upgrade to the latest version with:
uv add datacollective@latest
# or
pip install -U datacollectiveWe recommend taking a look at our package documentation for the full list of features and capabilities.
What do concurrent uploads mean to me?
Before the 0.7.0 update, the SDK split the file into chunks (10 MB by default) and uploaded each part one by one sequentially. Even if you were on a really high speed connection, that stream would rarely fill up, so the SDK wouldn't utilise the full potential of your connection. With the new release, these parts are now uploaded concurrently, which can lead to significant speed ups. You don't need to change anything in your code to benefit from this. Just upgrade to the latest version of our package.
How much faster?
These numbers were measured while developing the feature, on the same machine and a 300 Mbps connection for both versions:
Before (version 0.6.4) - Sequential:
For a 100 MB dataset, the average upload speed was 5.2 MB/s finishing at 21 sec.
For a 1 GB dataset, the average upload speed was 5.8 MB/s finishing at 2min 45 sec.
After (version 0.7.0) - 4 parallel workers:
For the same 100 MB dataset, the average upload speed was 14.9 MB/s finishing at 7 seconds, leading to a 2.6x speedup.
For the 1 GB dataset, the average upload speed was 21.4 MB/s finishing at 44 seconds, leading to a 3.6x speedup.
Tuning it to your needs
In a bit more technical terms, every upload function of the SDK (create_submission_with_upload, upload_dataset_file and upload_sample_file) now takes a max_workers argument, and it defaults to 4:
from datacollective import create_submission_with_upload
response = create_submission_with_upload(
file_path="my_dataset.tar.gz",
submission=submission,
max_workers=4, # number of parts uploaded at the same time
)You can tune the max_workers argument, however, not every use case will benefit equally from a high number of workers. In summary:
- Fast, stable connection, large archive: you can try raising the number of
max_workers=8, as well as increasing the chunk size of the parts being uploadedpart_size=50 * 1024 * 1024(50 MB). - Slow or flaky connection: keep parts small so a failure only re-uploads a little, and consider
max_workers=2. - Want the old behaviour:
max_workers=1uploads parts one at a time, exactly as before.
For more details on how this feature works and how to tune it to you needs, check out this section in our documentation.
Other interesting features
- Browse and search the catalogue without an API key!
list_datasetslists or searches our dataset catalogue with free-text queries and filters such as task, locale, license, pricing and whether a sample is available, with paginated results you can iterate over.list_dataset_filtersreturns the exact filter values currently accepted.
from datacollective import list_datasets, Task
page = list_datasets(
"swahili speech",
task=[Task.ASR, Task.TTS],
sort="newest",
results_per_page=20,
)
print(f"{page.total} matches")
for dataset in page:
print(dataset.id, dataset.name, dataset.locale, dataset.sizeBytes)Results are pydantic models (DatasetList & DatasetFilter) so that you can have typed access to the fields
- Download a dataset's sample:
download_samplefetches the small excerpt a publisher provides, so you can inspect the format before pulling the full archive. Samples are kept undersamples/<dataset-id>/in your download directory.
from datacollective import download_sample
sample_path = download_sample("your-dataset-id")- Delete a submission:
delete_submissionpermanently removes a submission that has not been published yet, together with its uploaded files. Handy for cleaning up after a test upload.
from datacollective import delete_submission
delete_submission(submission_id="your-submission-id")Next steps
If you publish datasets, the next upload you run will already be faster. If you download datasets, check whether the one you are interested in has a sample and give download_sample a try before pulling the full archive. Note that samples are also available for free for compensated datasets, so you can get a glimpse before purchasing. As always, bug reports and feature requests are welcome on the GitHub repository, and we welcome any type of feedback you might have at our address: [email protected].
