Storage of media files on Multiple Django or my first experience in Open source

In this article, I will try to share my first experience in open source, on how to build a python package by publishing it in PyPI, as well as to offer a solution for storing media files in scalable Django projects.

One day I was assigned to a ticket stating that you need to add an image field in one of the project’s models. Easy-peasy, I thought. I added an ImageField, updated a form, committed and pushed. Guys started to test and faced with the fact that the stored image after some time was not available. Our administrator was surprised and unhappy when he saw the media files in the project root. This is how I learned that a project I am working on is distributed across multiple Django instances.

Problem: The file is being downloaded from one Django instance, and the next time it is accessed, the user can get to the other Django instance and thus get 404 Not Found error. Also this a bad case, if the project is located in docker container and each time you build a project everything is deleted.

Solution: To store the media files as a separate service.

Googling some time, I found an interesting open source project, called Minio — a service for storing objects, written in Go. It has a pretty good documentation. there is an API implementation in Python and docker support. But there was no ready implementation for Django.

Thus I came up with an idea to kill two birds with one stone — to solve the problem in the project and finally try to create my open source project.

To store the files somewhere outside of the project, it is necessary to redefine the Default File Storage, using the Storage class. First of all, the class should be able to connect to the server minio.

Custom file storage class

Next, we need to implement some methods to save the file and obtain a reference to the file. Thus it is necessary to consider some nuances:

  • Check for the existence of the bucket (the place where the files are stored in minio service);
  • Check for the existence of a file in the bucket;
  • Method to get the file size, so it may be necessary for validation in the django form.

The rest of the code can be found in the project repository. How to deploy minio can be found here.

Creating and Publishing the Package

Okay. The code is written. Now it is important to make it accessible to the others. PyPI will help us with that. It is a repository that stores information about the programs and their releases. It’s the place where packages come from when you are using pip package manager. To do this, you need to prepare the package meta data and source code.

In setup.py file you should specify package information: name, version, dependencies, etc.

To prepare the sources use sdist command:

python setup.py sdist

Folder with archived sources, called dist will appear in the project root. Now you can upload the package in PyPI. Lets use twine for ease of use:

twine upload dist/my-package-0.1.tar.gz

To learn more about how the process of adding your package to PyPI can be found in the official tutorial.

Impressions From Making Open Source Software

While working on this problem I encountered some difficulties, but got very good support from Minio community. After the package was ready, the guys asked for permission to redirect the possible questions of compatibility with Django to me. Later, some guy, who is almost at the same time created a similar package wrote me and offered to work together.

It was a great experience for me, and I think it is a must have experience for each beginner developer. The sooner you try to make something on your own and make it open source, the more benefit you gain.

Special thanks to Minio developers and it’s community.

Software Development Services.

Mad Devs internship program automation

Mad Devs has an Internship program for different programming languages. The communication with interns takes place in Telegram messenger. 5...

Running Postgres as a build requirement in TeamCity build

We use different Continuous Integration tools in our projects. One of them is TeamCity software. A pipeline for TeamCity can be configured easily and...

What to choose to implement audio/video calls solution using WebRTC?

Mad Devs have been working with WebRTC since 2013. We have implemented several projects with video or audio calls enabled. That’s why I want to share...