Edit base.css file from S3 Bucket

I’m using AWS S3 to serve my static files – however I’ve just found out you can’t edit them directly from S3, which kind of makes it pointless as I will be continuously changing things on my website. So – is the conventional way to make the changes then re-upload the file? Or do most developers store their base.css file in their repository so it’s easier to change?

Because I’m using Django for my project so there is only supposed to be one static path (for me that’s my S3 bucket) – or is there another content delivery network where I can directly edit the contents of the file on the go which would be better?


Yes, imo it would be unusual to edit the files of your production website directly from where they are served.

Edit them locally, check them into your repo and then deploy them to s3 from your repo, perhaps using a tool like Jenkins. If you make a mistake, you have something to roll back to.

I can’t think of any circumstances where editing your files directly in production is a good idea.


How can I delete a repeated dictionary in list?

for dynamic values sometimes the value will be keep repeating, say if a variable

table = [

here {'man':'tim','age':'2','h':'5','w':'40'} dictionary set repeat twice these are dynamic value.

How can I stop repeating this, so list will not contain any repeated dictionary before rendering it to templates?

edited: actual data

[{'scorecardid': 1, 'progress2': 'preview', 'series2': 'Afghanistan v Zimbabwe in UAE, 2018', 'Commentary1': '/Commentary1', 'commentaryid': 1, 'matchid2': '10', 'matchno2': '5th ODI', 'teams2': 'AFG vs ZIM', 'matchtype2': 'ODI', 'Scorecard1': '/Scorecard1', 'status2': 'Starts on Feb 19 at 10:30 GMT'}, {'six2': '0', 'scorecardid': 2, 'overs5': '4', 'fours1': '0', 'overs10': '20', 'Batting_team_img': 'images/RSA.png', 'wickets20': '5', 'wickets6': '1', 'Bowling_team_img': 'images/IND.png', 'maidens6': '0', 'Batting team': 'RSA', 'matchid2': '9', 'name6': 'Unadkat', 'teams2': 'RSA vs IND', 'wickets10': '9', 'desc10': 'Inns', 'runs5': '32', 'matchtype2': 'T20', 'Scorecard1': '/Scorecard2', 'runs1': '2', 'wickets5': '0', 'runs6': '33', 'runs2': '0', 'maidens5': '0', 'runs20': '203', 'name5': 'Bumrah*', 'progress2': 'complete', 'Commentary1': '/Commentary2', 'fours2': '0', 'series2': 'India tour of South Africa, 2017-18', 'name1': 'Junior Dala*', 'commentaryid': 2, 'matchno2': '1st T20I', 'six1': '0', 'overs6': '4', 'Bowling team': 'IND', 'balls2': '2', 'balls1': '3', 'name2': 'Shamsi', 'overs20': '20', 'runs10': '175', 'desc20': 'Inns', 'status2': 'Ind won by 28 runs'}, {'scorecardid': 3, 'overs5': '0.4', 'fours1': '0', 'overs10': '18.4', 'Batting_team_img': 'images/BAN.png', 'wickets20': '4', 'wickets6': '1', 'Bowling_team_img': 'images/SL.png', 'Batting team': 'BAN', 'matchid2': '6', 'name6': 'Shanaka', 'teams2': 'BAN vs SL', 'wickets10': '10', 'desc10': 'Inns', 'runs5': '3', 'matchtype2': 'T20', 'Scorecard1': '/Scorecard3', 'runs1': '1', 'wickets5': '2', 'runs6': '5', 'maidens5': '0', 'runs20': '210', 'progress2': 'complete', 'Commentary1': '/Commentary3', 'name5': 'Gunathilaka*', 'series2': 'Sri Lanka tour of Bangladesh, 2018', 'name1': 'Nazmul Islam', 'commentaryid': 3, 'matchno2': '2nd T20I', 'six1': '0', 'overs6': '1.5', 'Bowling team': 'SL', 'maidens6': '0', 'balls1': '1', 'overs20': '20', 'runs10': '135', 'desc20': 'Inns', 'status2': 'SL won by 75 runs'}, {'six2': '2', 'scorecardid': 4, 'overs5': '4', 'fours1': '1', 'overs10': '20', 'Batting_team_img': 'images/NZ.png', 'wickets20': '7', 'wickets6': '1', 'Bowling_team_img': 'images/ENG.png', 'maidens6': '0', 'Batting team': 'NZ', 'matchid2': '4', 'name6': 'Tom Curran', 'teams2': 'NZ vs ENG', 'wickets10': '4', 'desc10': 'Inns', 'runs5': '41', 'matchtype2': 'T20', 'Scorecard1': '/Scorecard4', 'runs1': '7', 'wickets5': '0', 'runs6': '32', 'runs2': '37', 'maidens5': '0', 'runs20': '194', 'name5': 'Chris Jordan*', 'progress2': 'complete', 'Commentary1': '/Commentary4', 'fours2': '2', 'series2': 'England, Australia, New Zealand T20I Tri-Series, 2018', 'name1': 'de Grandhomme*', 'commentaryid': 4, 'matchno2': '6th Match', 'six1': '0', 'overs6': '3', 'Bowling team': 'ENG', 'balls2': '30', 'balls1': '5', 'name2': 'Chapman', 'overs20': '20', 'runs10': '192', 'desc20': 'Inns', 'status2': 'Eng won by 2 runs'}, {'scorecardid': 5, 'overs5': '7.4', 'fours1': '3', 'runs20': '213', 'six2': '0', 'commentaryid': 5, 'Batting team': 'SAUS', 'matchid2': '18770', 'matchno2': '21st Match', 'wickets10': '3', 'overs10': '49.4', 'matchtype2': 'TEST', 'runs1': '26', 'overs6': '8', 'runs6': '39', 'runs2': '49', 'name1': 'Mennie*', 'name5': 'Daniel Fallins*', 'series2': 'Sheffield Shield, 2017-18', 'Commentary1': '/Commentary5', 'wickets6': '1', 'runs11': '281', 'six1': '0', 'runs10': '192', 'balls1': '58', 'overs11': '74.1', 'maidens5': '1', 'desc21': '1st Inns', 'status2': 'South Aus won by 7 wkts', 'runs5': '51', 'wickets11': '10', 'desc11': '1st Inns', 'desc20': '2nd Inns', 'wickets20': '10', 'wickets21': '10', 'teams2': 'NSW vs SAUS', 'balls2': '85', 'Scorecard1': '/Scorecard5', 'wickets5': '1', 'progress2': 'Result', 'runs21': '256', 'fours2': '6', 'desc10': '2nd Inns', 'name6': 'Stobo', 'maidens6': '1', 'Bowling team': 'NSW', 'name2': 'Ferguson', 'overs20': '68.4', 'overs21': '90.4'}, {'six2': '0', 'scorecardid': 6, 'overs5': '4', 'fours1': '0', 'overs10': '20', 'Batting_team_img': 'images/RSA.png', 'wickets20': '5', 'wickets6': '1', 'Bowling_team_img': 'images/IND.png', 'maidens6': '0', 'Batting team': 'RSA', 'matchid2': '19166', 'name6': 'Unadkat', 'teams2': 'RSA vs IND', 'wickets10': '9', 'desc10': 'Inns', 'runs5': '32', 'matchtype2': 'T20', 'Scorecard1': '/Scorecard6', 'runs1': '2', 'wickets5': '0', 'runs6': '33', 'runs2': '0', 'maidens5': '0', 'runs20': '203', 'name5': 'Bumrah*', 'progress2': 'Result', 'Commentary1': '/Commentary6', 'fours2': '0', 'series2': 'India tour of South Africa, 2017-18', 'name1': 'Junior Dala*', 'commentaryid': 6, 'matchno2': '1st T20I', 'six1': '0', 'overs6': '4', 'Bowling team': 'IND', 'balls2': '2', 'balls1': '3', 'name2': 'Shamsi', 'overs20': '20', 'runs10': '175', 'desc20': 'Inns', 'status2': 'Ind won by 28 runs'}]


Since your records do not appear to have a unique identifier to differentiate records, you will need to hash on all key-value pairs. This approach will work as long as you do not have nested mutable objects inside your dictionaries.

I’ll use an OrderedDict here to maintain order.

from collections import OrderedDict
             map(frozenset, map(dict.items, table)), None

[{'age': '2', 'h': '5', 'man': 'tim', 'w': '40'},
 {'age': '4', 'h': '3', 'man': 'jim', 'w': '20'},
 {'age': '24', 'h': '5', 'man': 'jon', 'w': '80'},
 {'age': '7', 'h': '4', 'man': 'tto', 'w': '49'}]

Here’s what’s going on:

  1. Convert each dictionary to a frozenset of tuples. frozensets are hashable.
  2. Hash each frozenset as a key into an OrderedDict. Duplicates are removed automatically.
  3. Retrieve keys and convert back into a list of dictionaries.

There are many ways to reproduce the algorithm described above. I’ve used the functional programming tool – map – which python offers.

How to fix Django AWS EC2 Gunicorn ExecStart ExecStop end error?

I am trying to point my AWS Route 53 Domain to my EC2 IPv4 Public IP for my Django app, but I’m running into some gunicorn issues. The strange thing is that I am getting a successful nginx configuration messages, but yet it doesn’t work. I’ve already created a record set on Route 53.

gunicorn.service: Service lacks both ExecStart= and ExecStop= setting. Refusing.


ALLOWED_HOSTS = ['', 'myapp.com']


Description=gunicorn daemon
ExecStart=/home/ubuntu/my_app/venv/bin/gunicorn --workers 3 --bind unix:/home/ubuntu/my_app/my_app.sock my_app.wsgi:application

Nginx Code:

server {
  listen 80;
  server_name my_app.com www.my_app.com;
  location = /favicon.ico { access_log off; log_not_found off; }
  location /static/ {
      root /home/ubuntu/my_app;
  location / {
      include proxy_params;
      proxy_pass http://unix:/home/ubuntu/my_app/my_app.sock;

Nginx Test is successful but yet app won’t run:

ubuntu@ip-175-228-35-250:~$ sudo nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful


Anyone that cares. I logged into the system and check it. Attempted to run gunicorn manually and it gives us a different error! It seem to be confusion between Gunicorn and using the venv. I reset the venv and reinstalled the requirements. System now attempts to load the application correctly. Turned on debugging, so PandaNinja can debug the remaining issue due the application now crashing.

File "/usr/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 515, in spawn_worker
  File "/usr/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 122, in init_process
  File "/usr/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 130, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/usr/lib/python2.7/dist-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/usr/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 65, in load
    return self.load_wsgiapp()
  File "/usr/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/usr/lib/python2.7/dist-packages/gunicorn/util.py", line 366, in import_app
ImportError: No module named travel_buddy.wsgi

What AWS service can help speed up slow Django view functions

I have a Django application with a Postgres DB that has kind of run away from my client as the database has millions of records. That being said there are some view functions that make several database queries, and because of that, it is glacier slow. Here is an example of one of my view functions It is kind of messy because I was playing around with things to look for query optimizations, my apologies.

def get_filters(request):
        post_data = request.body.decode("utf-8")
        post_data = json.loads(post_data)
    except Exception as e:
        logging.error('There was an issue getting filter post data')

        return HttpResponse(
            json.dumps({'filters': {'makes': [], 'models': [], 'years': [], 'transmissions': [], 'fuel_types': []}}),

    dealer = post_data['dealer'] if 'dealer' in post_data else None
    make_ids = post_data['make_ids'] if 'make_ids' in post_data else []
    model_ids = post_data['model_ids'] if 'model_ids' in post_data else []
    years = post_data['years'] if 'years' in post_data else []
    condition = post_data['condition'] if 'condition' in post_data else None
    price_min = post_data['price_min'] if 'price_min' in post_data else 0
    price_max = post_data['price_max'] if 'price_max' in post_data else 10000000

    # Catch Critical Error Where There Is No Dealer
    if not dealer:
        logging.error('Unable to find a Dealer')
        return HttpResponse(
            json.dumps({'filters': {'makes': [], 'models': [], 'years': [], 'transmissions': [], 'fuel_types': []}}),
            dealer = Dealer.objects.get(id=dealer)
        except Exception as e:
            return HttpResponse(
                json.dumps({'filters': {'makes': [], 'models': [], 'years': [], 'transmissions': [], 'fuel_types': []}}),

    current_year = datetime.datetime.now().year
    start_year = current_year - 30

    # First get the make filters
    vehicles = Vehicle.objects.filter(dealer=dealer)

    if years:
        vehicles = vehicles.filter(year__in=years)

    filtered_make_names = vehicles.values_list('vehicle_make__name', flat=True)
    filtered_makes = VehicleMake.objects.filter(name__in=filtered_make_names)

    makes_map = [{
            'name': make.name,
            'count': vehicles.filter(vehicle_make=make, dealer=dealer).count(),
            'id': make.id
        } for make in filtered_makes

    # Second get the model filters

    filtered_model_names = vehicles.values_list('vehicle_model__name', flat=True)
    filtered_models = VehicleMake.objects.filter(name__in=filtered_model_names)

    dealer_models = VehicleModel.objects.filter(

    new_dealer_models = VehicleModel.objects.filter(name__in=vehicles.values_list('vehicle_model__name', flat=True))

    if len(make_ids) > 0:
        dealer_models = dealer_models.filter(make__id__in=make_ids)

    # Get the actual filters
    year_map = [{
        'year': yr,
        'count': Vehicle.objects.filter(year=yr, dealer=dealer).count()
    } for yr in range(start_year, current_year) if Vehicle.objects.filter(year=yr, dealer=dealer).count() > 0][::-1]

    models_map = [{
        'name': model.name,
        'count': Vehicle.objects.filter(vehicle_model=model, dealer=dealer).count(),
        'id': model.id
    } for model in dealer_models]

    filter_map = {
        "makes": makes_map,
        "models": models_map,
        "years": year_map,
        "transmissions": [],
        "fuel_types": []

    return HttpResponse(json.dumps({'filters': filter_map}), content_type='application/json')

I want to launch and AWS EC2 instance, and migrate my code from the server I have now to there, and was wondering what AWS services I could use in conjunction with that to make those view functions faster, and why ? Does autoscaling assist with that, or does autoscaling only kick in when the CPU has hit a certain point ?


You need to figure out if the bottleneck is in your database layer, or in your web layer.

If the bottleneck is you database layer, than a bigger db instance on its own server or the introduction of a caching layer such as memcache or redis might be appropriate (django has plugs for both of these)

If the bottleneck is your website, than a combination of a load-balancer, multiple ec2 instances running you website, and an autoscaling group might be appropriate.

But first, you really need to figure out where the bottleneck is so you don’t spend time and money optimizing the wrong thing.

Update model fields based on POST data before save with Django Rest Framework

I’m using django-rest-framework and want to augment the posted data before saving it to my model as is normally achieved using the model’s clean method as in this example from the django docs:

class Article(models.Model):
def clean(self):
    # Don't allow draft entries to have a pub_date.
    if self.status == 'draft' and self.pub_date is not None:
        raise ValidationError(_('Draft entries may not have a publication date.'))
    # Set the pub_date for published items if it hasn't been set already.
    if self.status == 'published' and self.pub_date is None:
        self.pub_date = datetime.date.today()

Unfortunately a django-rest-framework Serializer does not call a model’s clean method as with a standard django Form so how would I achieve this?


From official docs:

The one difference that you do need to note is that the .clean() method will not be called as part of serializer validation, as it would be if using a ModelForm. Use the serializer .validate() method to perform a final validation step on incoming data where required.

There may be some cases where you really do need to keep validation logic in the model .clean() method, and cannot instead separate it into the serializer .validate(). You can do so by explicitly instantiating a model instance in the .validate() method.

def validate(self, attrs):
    instance = ExampleModel(**attrs)
    return attrs

Sphinx search issues

    1. Access sphinx database:
      The sphinx indexes can be accessed with the following command:
      mysql -P 9306 -h 0
      Execute show tables;
      This will display the indexes. To see the data in the index execute:
      select * from index_name;
    2. WARNING: Attribute count is 0: switching to none docinfo
      Add the following to sphinx.conf source configuration.
      sql_attr_string = title # will be stored but will not be indexed
    3. ERROR: duplicate attribute name
      Check that you do not have sql_attr and sql_field pointed to the same column in sphinx.conf
      If you want the column to be a field and attribute then add it in sql_field else add it as sql_attr
    4. query error: no field ‘first_name’ found in schema\x0
      Add the following in sphinx.conf
      sql_field_string = title # will be both indexed and stored
    5. Overrriding sphinx.conf settings with SphinXql in django:
      Add the following in your settings.py

      'index_params': {
      'type': 'plain',
      'charset_type': 'utf-8'
      'searchd_params': {
      'listen': '9306:mysql41',
      'pid_file': os.path.join(INDEXES['sphinx_path'], 'searchd.pid')
    6. ERROR 1064 (42000): index : fullscan requires extern docinfo
      Add the following in sphinx.conf in the index section:
      docinfo = extern


Inner classes – Inheritance and overriding their attributes

I’m trying to understand how Django uses python’s metaclasses for it’s database models (options) and came up with the following stripped down code snippet that should roughly mimic Django’s logic.

class DatabaseOptions(object):

    def __init__(self, opts):
        if opts:
            for key, val in opts.__dict__.items():
                if not key.startswith('__') and not callable(val):
                    setattr(self, key, val)

class MetaModel(type):

    def __new__(cls, name, bases, classdict):
        result = super().__new__(cls, name, bases, dict(classdict))

        opts = classdict.pop('DbMeta', None)
        if opts:
            setattr(result, '_db_meta', DatabaseOptions(opts))

        return result

class Model(object, metaclass=MetaModel):

    class DbMeta:
        database = 'default'
        migrate = True

class User(Model):

    class DbMeta(Model.DbMeta):
        database = 'user'

class GroupUser(User):

    class DbMeta(User.DbMeta):
        database = 'group_user'

Using the above code, I would expect the following output:

print(Model._db_meta.database)  # default
print(Model._db_meta.migrate)  # True

print(User._db_meta.database)  # user
print(User._db_meta.migrate)   # True

print(GroupUser._db_meta.database)  # group_user
print(GroupUser._db_meta.migrate)  # True

Instead I get the following exception

>>> python3 test.py 
Traceback (most recent call last):
  File "test.py", line 48, in <module>
    print(User._db_meta.migrate)   # True
AttributeError: 'DatabaseOptions' object has no attribute 'migrate'

My question would be why User.DbMeta does not inherit the migrate attribute from Model.DbMeta? Is there any solution for this kind of problem?


According to Daniel’s answer I came up with the following that worked for me:

class DatabaseOptions(object):

    def __init__(self, opts):
        if opts:
            for key in dir(opts):
                if not key.startswith('__'):
                    val = getattr(opts, key, None)
                    if not callable(val):
                        setattr(self, key, val)


This isn’t a question about inner classes specifically.

Class attributes are just that, attributes of the class itself. So, the __dict__ of Model.DbMeta contains “database” and “migrate”, but the one of User.DbMeta contains only “database” because that is the only attribute defined by that class.

However, those attributes are shown by dir(); you should probably iterate over the results of calling that on the class in DatabaseOptions rather than __dict__.