Make sure to check out Project FURY : https://github.com/fury-gl/fury

Hey Everyone! 
This week , more shaders

What did I do this week

The task assigned for this week was to explore more shader techniques which could be implemented using FURY and which would demonstrate the capability of FURY shader system. So i decided to work on implementing shading examples such as Gooch shading and reflection shader using textures.

Below are the outputs of the techniques i worked on :

The shader demos are available at :
https://github.com/lenixlobo/fury/tree/shader-demos

What's coming up next

The next week will involve working on more such demos which can demonstrate the capabilities of FURY

Did I get stuck anywhere

No issues were faced this week

The python package named numpy come with corrcoef function to return Pearson product-moment correlation coefficients. This method has a limitation in that it can compute the correlation matrix between two variables only. The full name is the Pearson Product Moment Correlation (PPMC). The PPMC is not able to tell the difference between dependent variables and independent variables. The

Docker is a containerization platform that gained immense popularity in the IT world as a tool that can contain an application and help to deploy it to multiple environments.

History

Before Docker, there were different approaches whose goal was to obtain a reliable development and deployment process, which means testing and putting the application into production as a reproducible unit where the same result is always achieved.

The leading way to isolate and organize applications along with their their dependencies was to place each and every application in its own virtual machine. Physical hardware runs multiple such environments and the whole process is called virtualization.

Containerization

Containerization is a type of virtualization which brings virtualization to the operating system level. Containerization brings abstraction to the operating system, while virtualization brings abstraction to the hardware.

• Containers are lightweight and typically faster than Virtual Machines
• Containers share relevant libraries and resources
• The boot up process is very fast since the applications run on the host kernel

Docker

Docker is a tool used to easily package, distribute and deploy applications with all its dependencies into a standardized unit called a Docker image. It provides the ability to create predictable environments which can be used for development, testing and deploying the application in multiple environments, including production.

Some Docker concepts:

• Image: an executable package that includes everything needed to run the application
• Container: a runtime instance of an image

A Docker container can be connected with other containers that provide functionality to the application like a database or a load balancer.

In Zato we create Docker images of two kinds:

• A Zato quickstart image
• An image to deploy to cloud environments for Swarm, Kubernetes and Helm

The idea behind the Zato quickstart image is to very quickly provision a fully working all-in-one Zato cluster that can be used to test a new service locally or explore Zato but, because this is a real Zato cluster, it can also be used for production purposes.

The image used for cloud environments provides more flexibility and can be used to deploy an instance of a Zato component like server, scheduler or web admin into the Zato cluster, each under its own container. That is, whereas Zato quickstart contains all the Zato components in a single image, the cloud one splits the components into separate Docker containers.

Wrapping up

This is it for now - we will be talking about these two types of Zato images and how to use them in upcoming articles.

These are the ten most rated questions at Stack Overflow last week.
Between brackets: [question score / answers count]
Build date: 2020-08-08 21:20:26 GMT

1. Why is this regex so slow in java - [24/4]
2. Apply function to pandas row-row cross product - [10/5]
3. How can I count a pandas dataframe over duplications - [7/4]
4. Sum of consecutive pairs in a list including a sum of the last element with the first - [6/9]
5. Pygame loop checking velocity rarely - [6/2]
6. How do I know which versions of dependencies my application supports? - [6/2]
7. Pandas v1.1.0: Groupby rolling count slower than rolling mean & sum - [6/0]
8. Google cloud storage python client AttributeError: 'ClientOptions' object has no attribute 'scopes' occurs after deployment - [5/2]
9. Can't use bot.run('TOKEN') - Discord.py - [5/1]
10. Why does wrapping the random method of random.Random seem to affect the RNG? - [5/1]

What I have done this week

Works towards analyzing multistage dockerfile. I combined the draft PR and the review from my mentors, the new commit is the first step of my plan. We split the multistage dockerfile into seperate dockefiles for build. Here are the changes in the new commit.

1. Modified function check_multistage_dockerfile() to return.

2. Remove function split_multistage_dockerfile() since we are working on the building stage. split_multistage_dockerfile() can be improved on analyze stage.

To Do

1. Improve readability for function check_multistage_dockerfile().

2. Try build images and analyze on them.

Did I get stuck somewhere?

Not yet.

Send WhatsApp message/media using python.

In this article, we shall implement a basic version of a Turing Machine in python and write a few simple programs to execute them on the Turing machine.  This article is inspired by the edX / MITx course Paradox and Infinity and few of the programs to be executed on the (simulated) Turing machine are … Continue reading Simulating a Turing Machine with Python and executing programs on it

pytest 6 is out. Specifically, 6.0.1, as of July 31.

And there's lots to be excited about.
Anthony Sottile joins the show to discuss features, improvements, documentation updates and more.

Full release notes / changelog

Some of what we talk about:

• How to update (at least, how I do it)
  • Run your test suites with 5.4.3 or whatever the last version you were using
  • Update to 6
  • Run again. Same output? Probably good.
  • If there are any warnings, maybe fix those.
  • You can also run with pytest -W error to turn warnings into errors.
  • Then find out all the cool stuff you can do now
• New Features
  • pytest now supports pyproject.toml files for configuration.
    • but remember, toml syntax is different than ini files. mostly quotes are needed
  • pytest now includes inline type annotations and exposes them to user programs.
    • Most of the user-facing API is covered, as well as internal code.
  • New command-line flags --no-header and --no-summary
  • A warning is now shown when an unknown key is read from a config INI file.
    • The --strict-config flag has been added to treat these warnings as errors.
  • New required_plugins configuration option allows the user to specify a list of plugins, including version information, that are required for pytest to run. An error is raised if any required plugins are not found when running pytest.
• Improvements
  • You can now pass output to things like less and head that close the pipe passed to them.
    • thank you!!!
  • Improved precision of test durations measurement.
    • use --durations=10 -vv to capture and show durations
  • Rich comparison for dataclasses and attrs-classes is now recursive.
  • pytest --version now displays just the pytest version,
    • while pytest --version --version displays more verbose information including plugins.
  • --junitxml now includes the exception cause in the message XML attribute for failures during setup and teardown.
• Improved Documentation
  • Add a note about --strict and --strict-markers and the preference for the latter one.
  • Explain indirect parametrization and markers for fixtures.
• Bug Fixes
• Deprecations
• Trivial/Internal Changes
• Breaking Changes you might need to care about before upgrading
  • PytestDeprecationWarning are now errors by default.
    • Check the deprecations and removals page if you are curious.
  • -k and -m internals were rewritten to stop using eval(), this results in a few slight changes but overall makes them much more consistent
  • testdir.run().parseoutcomes() now always returns the parsed nouns in plural form.
    • I'd say that's an improvement

Special Guest: Anthony Sottile.

Sponsored By:

• Datadog: Modern monitoring & security. See inside any stack, any app, at any scale, anywhere. Visit testandcode.com/datadog to get started.

Support Test & Code : Python Testing for Software Engineering

Links:

• pytest Changelog / Release Notes
• Deprecations and Removals — pytest documentation

We have been spoiled to have Tim Ruscica, founder and star of the very informative and popular “Tech with Tim” YouTube channel, work on our team as a Software Engineering intern this summer! During his time here he implemented several features for our extension, including the much desired export notebook to HTML and PDF, as well as influence the discussion and design of other features we have been actively working on and shipping. We’re honored to have him spotlight a few of his favorite VS Code Python Extension features (not just those he worked on) in a recent video! Please check it out and give Tim some much deserved kudos!

If you’re interested in learning about something specific, follow the timestamps below!

00:22 – VSCode Python Setup

00:50 – Jupyter Notebooks

01:38 – Interactive Window

03:28 – Variable Explorer

04:41 – Run By Line

05:50 – Dataframe Viewer

06:10 – Notebook Exports

06:56 – Gather

08:28 – Pylance

Did Tim’s video give you an idea for a feature? Let us know here!

The post Need an Intro to VS Code? Let Tech with Tim Help! appeared first on Python.

Introduction

H2 is a lightweight database server written in Java. It can be embedded in Java applications, or run as a standalone server.

In this tutorial, we'll review why H2 can be a good option for your projects. We'll also learn how to integrate H2 with Python by building a simple Flask API.

The Features of H2

H2 was built with performance in mind.

"H2 is a combination of: fast, stable, easy to use, and features".

Although H2 is prominent mainly because it can be embedded in Java applications, it has some interesting features that also apply to its server version. Let's see some of them next.

Size and Performance

The .jar file used for the server version is around 2MB. We can download it from the H2 site, bundled with extra scripts and documentation. If we search in Maven Central though, we can download the .jar file on its own.

H2 performance shines in its embedded version. Even so, the official benchmark shows that its client-server version is also impressive.

In-Memory Databases and Encryption

In-Memory databases are not persistent. All data is stored in memory, so speed is greatly increased.

The H2 site explains that In-Memory databases are particularly useful when prototyping, or when using read-only databases.

Encryption is another useful feature to protect data at rest. Databases can be encrypted with the AES-128 algorithm.

Other Useful Features

H2 also provides a cluster mode, the ability to run multiple servers and connect them together. Writes are done in all servers at the same time, while reads are done from the first server in the cluster.

H2 surprises for its simplicity. It provides several useful features and it's easy to set up.

Let's start an H2 server in preparation for the following sections:

$ java -cp ./h2-1.4.200.jar org.h2.tools.Server -tcp -tcpAllowOthers -tcpPort 5234 -baseDir ./ -ifNotExists

The arguments that start with tcp enable communication to the server. The ifNotExists argument allows the database to be created when accessing it for the first time.

Description of the API and General Diagram

Let's suppose we are writing an API to register all exoplanets found to date. Exoplanets are planets found outside our Solar System, orbiting other stars.

This is our simple API definition, a CRUD for one resource:

This definition along with the rest of the code we'll see next is available in this GitHub repo.

This is how our application will look like at the end of this tutorial:

Left of the diagram we see the API Client. That client can be the "Try it out" function of the Swagger Editor, or any other client, like Postman or cURL.

On the other end we find the H2 database server, running on TCP port 5234 as explained above.

Finally, our application in the middle is composed of three Python files. The first one will have the Flask app that will answer all REST API requests. All endpoints we described in the definition above will be added to this file.

The second file will have the persistence, functions that access the database to execute the CRUD operations, using the JayDeBeApi package.

Lastly, a third file will contain a schema representing the resource the API manages, the Exoplanet. We'll use the Marshmallow package to represent that schema. The first two python files will use this schema to represent resources and pass them to each other.

Let's start off with the persistence file.

Database Schema

To store the Exoplanet resource to an H2 database we should write the basic CRUD functions first. Let's start by writing the creation of the database. We use the JayDeBeApi package to access databases through JDBC:

import jaydebeapi

def initialize():
    _execute(
        ("CREATE TABLE IF NOT EXISTS exoplanets ("
         "  id INT PRIMARY KEY AUTO_INCREMENT,"
         "  name VARCHAR NOT NULL,"
         "  year_discovered SIGNED,"
         "  light_years FLOAT,"
         "  mass FLOAT,"
         "  link VARCHAR)"))

def _execute(query, returnResult=None):
    connection  = jaydebeapi.connect(
        "org.h2.Driver",
        "jdbc:h2:tcp://localhost:5234/exoplanets",
        ["SA", ""],
        "../h2-1.4.200.jar")
    cursor = connection.cursor()
    cursor.execute(query)
    if returnResult:
        returnResult = _convert_to_schema(cursor)
    cursor.close()
    connection.close()

    return returnResult

The initialize() function is simple enough because of the helper functions after. It creates the exoplanets table if it doesn't exist already. This function should be executed before our API starts receiving requests. We'll see later where to do that with Flask.

The _execute() function contains the connection string and credentials to access the database server. It is simpler for this example, but there is room for improvement regarding security. We could save our credentials elsewhere, like environment variables for example.

Also, we added the path to the H2 jar file to the connect() method, as it has the driver we need to connect to H2 - org.h2.Driver.

The JDBC connection string ends in /exoplanets. This means that when connecting for the first time a database called exoplanets will be created.

You may have noticed that _execute() can return the result of the SQL query using the _convert_to_schema() function. Let's now see how that function works.

Marshmallow Schemas and CRUD Database Functions

Some SQL queries return tabular results, particularly the SELECT statement. JayDeBeApi will format those results as a list of tuples. For example, for the schema defined in the last section we could get a result similar to this:

>>> connection  = jaydebeapi.connect(...
>>> cursor = connection.cursor()
>>> cursor.execute("SELECT * FROM exoplanets")
>>> cursor.fetchall()

[(1, 'Sample1', 2019, 4.5, 1.2, 'http://sample1.com')]

Nothing's stopping us from managing results in this format and eventually return it to the API client. But looking ahead we know we will use Flask, so it would be good to already return results in a format Flask recommends.

In particular, we'll be using Flask-RESTful to facilitate the use of API routes. That package recommends to use Marshmallow to parse requests. This step allows for normalization of the objects. This way we can discard unknown properties and highlight validation errors, for example.

Let's see how the Exoplanet class would look so we can discuss further:

from marshmallow import Schema, fields, EXCLUDE

class ExoplanetSchema(Schema):
    id = fields.Integer(allow_none=True)
    name = fields.Str(required=True, error_messages={"required": "An exoplanet needs at least a name"})
    year_discovered = fields.Integer(allow_none=True)
    light_years = fields.Float(allow_none=True)
    mass = fields.Float(allow_none=True)
    link = fields.Url(allow_none=True)
    class Meta:
        unknown = EXCLUDE

The definition of the properties looks familiar. It's the same as the database schema, including the definition of required fields. All fields have a type that defines some default validation. For example, the link field is defined as an URL, so a string that doesn't look like an URL won't be valid.

Specific error messages can also be included here, like the validation for a name.

For this example project, we want to discard, or exclude, all unknown fields and API client may send erroneously. This is achieved in the Meta nested class.

Now we can use the load() and loads() Marshmallow methods to convert and validate our resources.

Now that we're acquainted with Marshmallow, we can explain what the _convert_to_schema() does:

def _convert_to_schema(cursor):
    column_names = [record[0].lower() for record in cursor.description]
    column_and_values = [dict(zip(column_names, record)) for record in cursor.fetchall()]

    return ExoplanetSchema().load(column_and_values, many=True)

In JayDeBeApi, column names are saved in the description field of the cursor, while the data can be obtained with the fetchall() method. We used list comprehensions in the first two lines to get the column names and values, and zip() to merge them.

The last line takes the merged result and converts them to ExoplanetSchema objects that Flask can further process.

Now that we explained the _execute() function and the ExoplanetSchema class, let's see all the CRUD database functions:

def get_all():
    return _execute("SELECT * FROM exoplanets", returnResult=True)

def get(Id):
    return _execute("SELECT * FROM exoplanets WHERE id = {}".format(Id), returnResult=True)

def create(exoplanet):
    count = _execute("SELECT count(*) AS count FROM exoplanets WHERE name LIKE '{}'".format(exoplanet.get("name")), returnResult=True)
    if count[0]["count"] > 0:
        return

    columns = ", ".join(exoplanet.keys())
    values = ", ".join("'{}'".format(value) for value in exoplanet.values())
    _execute("INSERT INTO exoplanets ({}) VALUES({})".format(columns, values))

    return {}

def update(exoplanet, Id):
    count = _execute("SELECT count(*) AS count FROM exoplanets WHERE id = {}".format(Id), returnResult=True)
    if count[0]["count"] == 0:
        return

    values = ["'{}'".format(value) for value in exoplanet.values()]
    update_values = ", ".join("{} = {}".format(key, value) for key, value in zip(exoplanet.keys(), values))
    _execute("UPDATE exoplanets SET {} WHERE id = {}".format(update_values, Id))

    return {}

def delete(Id):
    count = _execute("SELECT count(*) AS count FROM exoplanets WHERE id = {}".format(Id), returnResult=True)
    if count[0]["count"] == 0:
        return

    _execute("DELETE FROM exoplanets WHERE id = {}".format(Id))
    return {}

All functions are mainly SQL queries, but create() and update() deserve some more explanation.

The INSERT SQL statement can receive column and values separated, in the form INSERT INTO table (column1Name) VALUES ('column1Value'). We can use the join() function to merge all columns and separate them with commas, and do something similar to join all values we want to insert.

The UPDATE SQL statement is a bit more complex. Its form is UPDATE table SET column1Name = 'column1Value'. So we need to alternate keys and values, and we did it using the zip() function.

All these functions return None when there is a problem. Later when we call them we will have to check for that value.

Let's save all database functions on its own file, persistence.py, so we can add some context when we call the functions, like this:

import persistence

persistence.get_all()

REST API With Flask

Now that we wrote a layer to abstract the access to the database, we are ready to write the REST API. We'll use the Flask and Flask-RESTful packages to make our definition as easy as possible. As we learned before, we'll also use Marshmallow to validate resources.

Flask-RESTful requires to define one class per API resource, in our case the Exoplanet resource only. Then we can associate that resource with a route like this:

from flask import Flask
from flask_restful import Resource, Api

app = Flask(__name__)
api = Api(app)

class Exoplanet(Resource):
    # ...

api.add_resource(Exoplanet, "/exoplanets", "/exoplanets/<int:Id>")

This way all our routes, /exoplanets and /exoplanets/<int:Id> will be directed to the class we defined.

For example, the GET /exoplanets endpoint will be answered by a method called get() inside the Exoplanet class. Because we also have the GET /exoplanet/<Id> endpoint, that get() method must have an optional parameter called Id.

Let's see the whole class to understand this better:

from flask import request
from flask_restful import Resource, abort
from marshmallow import ValidationError
import persistence

class Exoplanet(Resource):
    def get(self, Id=None):
        if Id is None:
            return persistence.get_all()

        exoplanet = persistence.get(Id)
        if not exoplanet:
            abort(404, errors={"errors": {"message": "Exoplanet with Id {} does not exist".format(Id)}})
        return exoplanet

    def post(self):
        try:
            exoplanet = ExoplanetSchema(exclude=["id"]).loads(request.json)
            if not persistence.create(exoplanet):
                abort(404, errors={"errors": {"message": "Exoplanet with name {} already exists".format(request.json["name"])}})
        except ValidationError as e:
            abort(405, errors=e.messages)

    def put(self, Id):
        try:
            exoplanet = ExoplanetSchema(exclude=["id"]).loads(request.json)
            if not persistence.update(exoplanet, Id):
                abort(404, errors={"errors": {"message": "Exoplanet with Id {} does not exist".format(Id)}})
        except ValidationError as e:
            abort(405, errors=e.messages)

    def delete(self, Id):
        if not persistence.delete(Id):
            abort(404, errors={"errors": {"message": "Exoplanet with Id {} does not exist".format(Id)}})

The remaining HTTP verbs are processed in the same way as GET, by the methods with name post(), put() and delete() .

As we said before, logic errors when accessing the database will cause the functions to return None. Those errors are captured here when needed.

Also, exceptions that represent validation errors are triggered by Marshmallow, so those errors are also captured and returned to the user along with an appropriate return error.

Conclusion

H2 is a useful database server, performant and easy to use. Although it is a Java package, it can also run as a standalone server, so we can use it in Python with the JayDeBeApi package.

In this tutorial we defined a simple CRUD application to illustrate how to access the database, and which functions are available. After that, we defined a REST API with Flask and Flask-RESTful.

Although several concepts were omitted for the sake of brevity, like authentication and paging, this tutorial is a good reference to start using H2 in our Flask projects.

Do you want to learn the how and when of implementing K-means clustering in Python? Would you like to practice your pandas skills with a real-world project? This week on the show, David Amos is back with another batch of PyCoder’s Weekly articles and projects.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Hello everyone

I know I am late for the weekly blog. My apologies for that.
The previous week was my first week post my exams. I had mentioned in the last blog that I will be working on the documentation for panda3d.
So, just as I mentioned, I was able to complete the documentation part in one week. The documentation includes documentation for both python and c++, the languages supported by panda3d for game development. The appropriate code snippets have been included along with the screenshots of the corresponding output.
The challenges were to understand the reStructured text format by Sphinx ans write the code for documentation with proper identation and tags so as to build separate docs for python and c++.
You may find the Pull Request here: #71

This week I am working on the sample code. Some part of work has been done and hopefully it will be completed by the weekend. I will share more about the same in my next blog.

Thank you!

Stay Safe!

The post [Video] How to Learn Python? (From Zero to Hero) appeared first on Afternerd.

Leverage DLLs from Python. Send faxes with Python without writing any code to send faxes.

Django is one of the most popular Python Web Frameworks. However, optimizing django performance can often be difficult. Most performance problems boil down to one thing: the database. According to our guest speaker, Andrew Brookins, the only way to master Django performance is to master database performance.

In this Webinar, Andrew is going to take us on a journey through an app that he created to demonstrate all the bottlenecks that arise when trying to optimize your Django app.

• Monday, August 10
• 7:00PM CEST – 1:00PM Eastern Daylight Time
• Register here

Outline

• Querying
  • Tackling the N+1 Problem: Reducing the number of queries to make your pages faster.
  • Annotations: Getting your database to do the heavy lifting for you
  • Iterator(): Get Django to chuck responses from your database
• Indexing
  • Covering Indexes: Turning queries that take several hundred milliseconds to tens of milliseconds.
  • Materialized Views: One of the most powerful ways to have large queries created for you
• Caching with Redis
  • Using Redis as a Cache
  • Custom Redis Authentication

About the Speaker

Andrew Brookins has over a decade of experience with Django and Relational Databases. That experience spans DevOps, application development, and platform engineering. He has worked on ecommerce sites, mobile APIs, comic book readers, particle simulation systems, learning and volunteering apps, and now works on a Databases as a Service at Redis Labs.

Andrew has also authored "Temple of Django Database Performance" which dives into the even greater detail than this webinar does. Furthermore, he has recently released a course on "Redis for Python Developers" at Redis University.

Andrew lives in Portland, Oregon with his wife and two wonderful children.

We're back with another GeekOut episode. Richard Campbell, a developer and podcaster who also dives deep into science and tech topics, is back for our second GeekOut episode. Last time we geeked out about the real science and progress around a moon base. This time it's why is there life on Earth, where could it be or have been in the solar system, and beyond. <br/> <br/> In case you didn't catch the first GeekOut, episode 253, this one is more of a general science and tech episode. I love digging into the deep internals of all the tools of the Python space, but given all that is going on in the world, I thought it'd be fun to take a step back and just enjoy some fun geekery and give you all something to just sit back and let your mind dream.<br/> <br/> <strong>Links from the show</strong><br/> <br/> <div><b>Richard on Twitter</b>: <a href="https://twitter.com/richcampbell" target="_blank" rel="noopener">@richcampbell</a><br/> <b>All Richard's GeekOut Episodes</b>: <a href="http://geekout.show" target="_blank" rel="noopener">geekout.show</a><br/> <b>Moonbase Geekout Episode</b>: <a href="https://talkpython.fm/253" target="_blank" rel="noopener">talkpython.fm/253</a><br/> <b>High Altitude Venus Operational Concept (HAVOC)</b>: <a href="https://sacd.larc.nasa.gov/smab/havoc/" target="_blank" rel="noopener">sacd.larc.nasa.gov</a><br/> <b>New Horizons</b>: <a href="https://solarsystem.nasa.gov/missions/new-horizons/in-depth/" target="_blank" rel="noopener">solarsystem.nasa.gov</a><br/> <b>The Planets: Saturn - NOVA documentary (Cassini)</b>: <a href="https://www.pbs.org/wgbh/nova/video/the-planets-saturn/" target="_blank" rel="noopener">pbs.org</a><br/> <b>Mission to Jupiter - Galileo</b>: <a href="https://www.jpl.nasa.gov/missions/galileo/" target="_blank" rel="noopener">jpl.nasa.gov</a><br/></div><br/> <strong>Sponsors</strong><br/> <br/> <a href='https://talkpython.fm/brilliant'>Brilliant</a><br> <a href='https://talkpython.fm/training'>Talk Python Training</a>

<p>Sponsored by us! Support our work through:</p> <ul> <li>Our <a href="https://training.talkpython.fm/"><strong>courses at Talk Python Training</strong></a></li> <li><a href="https://t.co/AKfVKcveg6?amp=1"><strong>Brian’s pytest book</strong></a></li> </ul> <p><strong>Brian #1: Start using pip install --use-feature=2020-resolver if you aren’t already</strong></p> <ul> <li>Mathew Feickert</li> <li><p>You may see something interesting when you run pip <img src="https://pbs.twimg.com/media/EeERublXoAILPct?format=png&amp;name=small" alt="Image" /></p></li> <li><p>This is not a problem. Do not adjust your sets. </p></li> <li>But, you should be aware of it.</li> <li>Especially if you install from requirements generated with <code>pip freeze</code>, you’ll want to use <code>--use-feature=2020-resolver</code> everywhere:</li> </ul> <pre><code> $ python -m pip install --use-feature=2020-resolver -r requirements_original.txt $ python -m pip freeze &gt; requirements_lock.txt $ python -m pip install --use-feature=2020-resolver -r requirements_lock.txt </code></pre> <ul> <li>Otherwise, you may run into issues <ul> <li>see <a href="https://gist.github.com/matthewfeickert/78cd03376435c05b6104f376cd43b537">Example of --use-feature=2020-resolver breaking a pip freeze requirements.txt</a></li> <li>and <a href="https://pip.pypa.io/en/latest/user_guide/#changes-to-the-pip-dependency-resolver-in-20-2-2020">Changes to the pip dependency resolver in 20.2 (2020)</a></li> </ul></li> </ul> <p><strong>Michael #2:</strong> <a href="https://twitter.com/mkennedy/status/1276606467156537344"><strong>Profiling Python import statements</strong></a></p> <ul> <li>Conversation with Brandon Braner lead to <a href="https://pypi.org/project/import-profiler/">import-profiler</a></li> <li>A basic python import profiler to find bottlenecks in import times. </li> <li>Not often a problem, imports can be an issue for applications that need to start quickly, such as CLI tools. </li> <li>Goal of import profiler is to help find the bottlenecks when importing a given package.</li> <li>Example</li> </ul> <pre><code> from import_profiler import profile_import with profile_import() as context: # Anything expensive in here import requests # Print cumulative and inline times. The number of + in the 3rd column # indicates the depth of the stack. context.print_info() </code></pre> <p>Output:</p> <pre><code> umtime (ms) intime (ms) name 83 0.5 requests 55 0.5 +packages.urllib3.contrib 54.1 0.3 ++ 53.1 0.7 +++connectionpool 6.3 1.1 ++++logging ... </code></pre> <p><strong>Brian #3:</strong> <a href="https://www.mattlayman.com/blog/2020/django-testing-toolbox/"><strong>Django Testing Toolbox</strong></a></p> <ul> <li>Matt Layman</li> <li>Testing packages commonly used on Django projects</li> <li>Techniques Matt uses</li> <li>packages <ul> <li>pytest-django - duh</li> <li>factory_boy - fake data</li> <li>django-test-plus - beefed up TestCase with tons of helper utilities</li> </ul></li> <li>techniques <ul> <li>Using TestCase and test classes instead of functions</li> <li>Arrange, Act Assert structure</li> <li>In-memory SQLite database</li> <li>Disable migrations while testing</li> <li>faster password hasher </li> <li>Use your editor effectively to run tests</li> </ul></li> </ul> <p><strong>Michael #4:</strong> <a href="https://github.com/pandas-profiling/pandas-profiling"><strong>Pandas-profiling</strong></a></p> <ul> <li>Recommended by Oz</li> <li>Considering the fact that Data Science users are almost the majority now for Python, I thought it would be nice to spread the word of pandas-profiling package. </li> <li>This package enables you to do Exploratory Data Analysis (EDA) with a single command. It is really useful to understand what percent of the data is NULL, or how it is distributed. </li> <li><p>I used to do these steps manually and it was quite time consuming. Now it is a breeze.</p></li> <li><p>Generates profile reports from a pandas <code>DataFrame</code>. The pandas <code>df.describe()</code> function is great but a little basic for serious exploratory data analysis. <code>pandas_profiling</code> extends the pandas DataFrame with <code>df.profile_report()</code> for quick data analysis.</p></li> <li>Features <ul> <li><strong>Type inference</strong>: detect the <a href="https://github.com/pandas-profiling/pandas-profiling#types">types</a> of columns in a dataframe.</li> <li><strong>Essentials</strong>: type, unique values, missing values</li> <li><strong>Quantile statistics</strong> like minimum value, Q1, median, Q3, maximum, range, interquartile range</li> <li><strong>Descriptive statistics</strong> like mean, mode, standard deviation, sum, median absolute deviation, coefficient of variation, kurtosis, skewness</li> <li><strong>Most frequent values</strong></li> <li><strong>Histogram</strong></li> <li><strong>Correlations</strong> highlighting of highly correlated variables, Spearman, Pearson and Kendall matrices</li> <li><strong>Missing values</strong> matrix, count, heatmap and dendrogram of missing values</li> <li><strong>Text analysis</strong> learn about categories (Uppercase, Space), scripts (Latin, Cyrillic) and blocks (ASCII) of text data.</li> <li><strong>File and Image analysis</strong> extract file sizes, creation dates and dimensions and scan for truncated images or those containing EXIF information.</li> </ul></li> <li>Nice examples too <ul> <li><a href="https://pandas-profiling.github.io/pandas-profiling/examples/master/census/census_report.html">Census Income</a> (US Adult Census data relating income)</li> <li><a href="https://pandas-profiling.github.io/pandas-profiling/examples/master/meteorites/meteorites_report.html">NASA Meteorites</a> (comprehensive set of meteorite landings) </li> <li><a href="https://pandas-profiling.github.io/pandas-profiling/examples/master/titanic/titanic_report.html">Titanic</a> (the "Wonderwall" of datasets) </li> <li><a href="https://pandas-profiling.github.io/pandas-profiling/examples/master/nza/nza_report.html">NZA</a> (open data from the Dutch Healthcare Authority)</li> <li><a href="https://pandas-profiling.github.io/pandas-profiling/examples/master/stata_auto/stata_auto_report.html">Stata Auto</a> (1978 Automobile data)</li> <li><a href="https://pandas-profiling.github.io/pandas-profiling/examples/master/vektis/vektis_report.html">Vektis</a> (Vektis Dutch Healthcare data)</li> <li><a href="https://pandas-profiling.github.io/pandas-profiling/examples/master/colors/colors_report.html">Colors</a> (a simple colors dataset)</li> </ul></li> </ul> <p><strong>Brian #5:</strong> <a href="https://rednafi.github.io/digressions/python/2020/07/03/python-mixins.html"><strong>Interfaces, Mixins and Building Powerful Custom Data Structures in Python</strong></a></p> <ul> <li>Redowan Delowar</li> <li>“Supercharging Python's built-in data structures”</li> <li>Discussion of interfaces, abstract base classes, mixins, and Python</li> <li>Class hierarchies and utilizing interfaces, ABCs (informal and formal), mixins, etc. are less common in Python than in some other languages, but you can still use them to do some really cool things.</li> <li>I especially liked the introduction to the concepts and why they are useful, as well as why ABCs are different than interfaces.</li> <li>“Interfaces can be thought of as a special case of Abstract Base Classes</li> </ul> <blockquote> <p>"It’s imperative that all the methods of an interface are abstract methods and the classes don’t store any state (instance variables). However, in case of abstract base classes, the methods are generally abstract but there can also be methods that provide implementation (concrete methods) and also, these classes can have instance variables."</p> </blockquote> <ul> <li>Shows an example, and goes on to discuss mixins, a parent class that provides functionality to subclasses but is not intended to be instantiated itself.</li> </ul> <p><strong>Michael #6:</strong> <a href="https://nedbatchelder.com/blog/202006/pickles_nine_flaws.html"><strong>Pickle’s 9 flaws</strong></a></p> <ul> <li>Python’s pickle module is a very convenient way to serialize and de-serialize objects. It needs no schema, and can handle arbitrary Python objects. But it has problems. This post briefly explains the problems.</li> <li>Via <a href="https://pycoders.com/">pycoders.com</a></li> <li>Article by Ned Batchelder <ul> <li><strong>Insecure</strong> - The insecurity is not because pickles contain code, but because they create objects by calling constructors named in the pickle. Any callable can be used in place of your class name to construct objects.</li> <li><strong>Old pickles look like old code</strong> - if your code changes between the time you made the pickle and the time you used it, your objects may not correspond to your code.</li> <li><strong>Implicit</strong> - they will serialize whatever structure your object has.</li> <li><strong>Over-serializes</strong> - They serialize everything in your objects, even data you didn’t want to serialize. For example, you might have an attribute that is a cache of computation that you don’t want serialized.</li> <li><strong>__init__ isn’t called</strong> - Pickles store the entire structure of your objects. When the pickle module recreates your objects, it does not call your __init__ method, since the object has already been created.</li> <li><strong>Python only</strong> - Pickles are specific to Python, and are only usable by other Python programs.</li> <li><strong>Unreadable</strong> - A pickle is a binary data stream (actually instructions for an abstract execution engine.) If you open a pickle as a plain file, you cannot read its contents.</li> <li><strong>Appears to pickle code</strong> - Functions and classes are first-class objects in Python: you can store them in lists, dicts, attributes, and so on. Pickle will gladly serialize objects that contain callables like functions and classes. But it doesn’t store the code in the pickle, just the name of the function or class.</li> <li><strong>Slow</strong> - Compared to other serialization techniques, pickle can be slow as Ben Frederickson demonstrates in <a href="https://www.benfrederickson.com/dont-pickle-your-data/">Don’t pickle your data</a>.</li> </ul></li> </ul> <p>Extras:</p> <p>Michael: </p> <ul> <li>More on <code>Pathlib</code> by <a href="https://twitter.com/flutefreak7/status/1287200446847823874">Brett Ables:</a> <code>text = Path(file).read_text()</code></li> <li>I was a guest on <a href="https://twitter.com/aQoCode/status/1287826329203810310">A Question of Code podcast</a>. Ed, Tom, and I discussed:</li> </ul> <p>Why is coding in Python such fun? And why is it so good for beginners and experts alike? Why might Python give you tangible results faster than JavaScript? And once you've learnt some Python, what are your career options? Find out all this and more in this week's pythonic installment of A Question of Code.</p> <p><strong>Joke</strong></p> <ul> <li>(<a href="https://twitter.com/ab_was_taken/status/1287884897802006528">via Aaron Brown</a>)</li> <li>By Caitlin Hudon 👩🏼‍💻 <a href="https://twitter.com/beeonaposy/status/1287797396471193601">@beeonaposy</a>:</li> <li>I have a Python joke, but I don't think this is the right environment.</li> </ul>

The post Python: How to Convert a List into a String? (The Right Way) appeared first on Afternerd.

What I did this week?

I took a break for the week. So there's nothing on my list.

What is coming up next?

From my last work there were few minor things that need to be changed in my triage PR. I'll be working on adding a new format that is PDF. I need some time to figure out what is the best case to add this feature that is either using a different output format or providing a new template so the users can directly print from the HTML.  

Have I got stuck anywhere?

There are no blocking issues for me at this moment.

Hi everyone  a little late on this blog as I have to go back to college to get my stuff back. I passed the second evaluation, yay,. Till now I have been able to get the module metadata except the licenses. The licenses can be collected in 2 ways one is through making bash scripts to find and collect licenses from the module directory. Since licenses files dont have a set structure or a convention that's why its getting difficult and a foolproof  guarantee cant be given. Another way is through making an in house copyright to License parser. For debian licenses we have debut  used in tern I have to come up with similar kind of parser. I also have one idea of training a model from data collected by using github's api but thats just overkill.

The post Python: Sort vs Sorted appeared first on Afternerd.

What did I do till now?

Last week I completed the ScrapyH2ProxyAgent implementation and added the required tests. I was going through the codebase for hyper-h2 library to get insight on how they implemented CONNECT method for HTTP/2. 

What's coming up next?

Next week I plan to finish working on ScrapyTunnelingH2Agent which enables a user to create a SSL Tunnel and proxy requests.

Did I get stuck anywhere?

Yeah I am stuck at a weird problem where two test cases are colliding i.e none of them being related to each other but fails when I run them both together and passes when I run them separately. I'm still working on finding a working fix! 

Your server is running just fine, handling requests and sending responses. But then, ever so slowly, memory usage creeps up, and up, and up–until eventually your process runs out of memory and crashes. And then it restarts, and the leaking starts all over again.

In order to fix memory leaks, you need to figure out where that memory is being allocated. And that can be tricky, unless you use the right tools.

Let’s see how you can identify the exact lines of code that are leaking by using the Fil memory profiler.

In this episode, we did some Django model development. I created a new model to track break days in a school year. This model will be critical to fill in vacations and holidays so that the scheduling functionality works properly. I added the model, the tests, the admin page, and the create view to create break days in the app. We started by picking a model name and discussing naming in programming.