In this article, I will try to share my first experience in open source, on how to build a python package by publishing it in PyPI, as well as to offer a solution for storing media files in scalable Django projects.
One day I was assigned a ticket stating that I need to add an image field to one of the project’s models. Easy-peasy, I thought. I added an ImageField, updated a form, committed, and pushed. Guys started to test and were faced with the fact that the stored image after some time was not available. Our administrator was surprised and unhappy when he saw the media files in the project root. This is how I learned that a project I am working on is distributed across multiple Django instances.
Problem: The file is being downloaded from one Django instance, and the next time it is accessed, the user can get to the other Django instance and thus get a 404 Not Found error. Also, this is a bad case, if the project is located in a docker container, and each time you build a project, everything is deleted.
Solution: To store the media files as a separate service.
Googling some time, I found an interesting open source project called Minio — a service for storing objects, written in Go. It has pretty good documentation. there is an API implementation in Python and Docker support. But there was no ready implementation for Django.
Thus I came up with an idea to kill two birds with one stone — to solve the problem in the project and finally try to create my open source project.
To store the files somewhere outside of the project, it is necessary to redefine the Default File Storage using the Storage class. First of all, the class should be able to connect to the server minio.
Custom file storage class
Next, we need to implement some methods to save the file and obtain a reference to the file. Thus it is necessary to consider some nuances:
- Check for the existence of the bucket (the place where the files are stored in minio service);
- Check for the existence of a file in the bucket;
- Method to get the file size, so it may be necessary for validation in the django form.
The rest of the code can be found in the project repository. How to deploy minio can be found here.
Creating and publishing the package
Okay. The code is written. Now it is important to make it accessible to the others. PyPI will help us with that. It is a repository that stores information about the programs and their releases. It’s the place where packages come from when you are using pip package manager. To do this, you need to prepare the package meta data and source code.
In setup.py file you should specify package information: name, version, dependencies, etc.
To prepare the sources use sdist command:
python setup.py sdist
Folder with archived sources, called dist will appear in the project root. Now you can upload the package in PyPI. Lets use twine for ease of use:
twine upload dist/my-package-0.1.tar.gz
To learn more about how the process of adding your package to PyPI can be found in the official tutorial.
Impressions from making Open Source software
While working on this problem I encountered some difficulties, but got very good support from Minio community. After the package was ready, the guys asked for permission to redirect the possible questions of compatibility with Django to me. Later, some guy, who is almost at the same time created a similar package wrote me and offered to work together.
It was a great experience for me, and I think it is a must have experience for each beginner developer. The sooner you try to make something on your own and make it open source, the more benefit you gain.
Special thanks to Minio developers and it’s community.
