Simple Web Page to Read Write Config File
From Novice to Proficient: How to Write a Configuration file in Python
Care for config file like your product code
Westwardhen nosotros pattern software, we normally put a lot of effort into writing high-quality code. Simply that'south not plenty. A expert software should too take care of its eco-system, like testing, deployment, network, etc. I of the virtually of import aspects is configuration management.
Proficient configuration direction should allow the software to be executed in whatsoever environment without changing the code. It helps Ops to manage all the hassle settings and it provides a view on what can happen during the process and even allows them to change the behavior during the runtime.
The most common configuration includes credentials to the database or an external service, the hostname of the deployed server, dynamic parameters, etc.
In this article, I want to share with you some good practices of configuration direction and how we tin implement them in Python. If y'all take more ideas, please go out your comments below.
When do we need a separate configuration file?
Before writing any configuration file, we should ask ourselves why nosotros need an external file? Tin't we just make them constants in the code? Actually, the famous The Twelve-Factor App has answered this question for united states:
A litmus test for whether an app has all config correctly factored out of the code is whether the codebase could exist fabricated open source at whatsoever moment, without compromising whatsoever credentials. Note that this definition of "config" does not include internal application config, such as
config/routes.rbin Runway, or how lawmaking modules are connected in Spring. This type of config does not vary between deploys, and and so is all-time washed in the lawmaking.
Information technology recommends that any environment-dependent parameters such as database credentials should sit in the external file. Otherwise, they are merely normal constants in the code. Another use instance I see a lot is to store dynamic variables in the external file, for instance, a blacklist or whitelist. Merely information technology tin also be a number within a certain range (due east.one thousand. timeout) or some free texts. These variables can possibly be the same in each environment, just the configuration file makes the software much more flexible and like shooting fish in a barrel to edit. Yet, if it grows besides much, we might consider moving it to a database instead.
Which format of the configuration file should I use?
In fact, there are no constraints on the format of the configuration file as long as the code could read and parse them. Only, in that location are some good practices.
The near common and standardized formats are YAML, JSON, TOML and INI. A good configuration file should encounter at to the lowest degree these 3 criteria:
- Piece of cake to read and edit: Information technology should be text-based and structured in such a way that is easy to understand. Even non-developers should be able to read.
- Allow comments: Configuration file is non something that will be simply read by developers. It is extremely of import in production when not-developers endeavor to empathise the process and change the software behavior. Writing comments is a fashion to chop-chop explain sure things, thus making the config file more expressive.
- Easy to deploy: Configuration file should be accepted by all the operating systems and environments. Information technology should also be easily shipped to the server via a CDaaS pipeline.
Maybe you withal don't know which one is better. But if you think nigh it in the context of Python, then the answer would be YAML or INI. YAML and INI are well accepted by most of the Python programs and packages. INI is probably the most straightforward solution with simply 1 level of the bureaucracy. Notwithstanding, there is no information type in INI, everything is encoded as a cord.
The same configuration in YAML looks similar this. As y'all can see, YAML supports nested structures quite well (like JSON). Too, YAML natively encodes some data types such as string, integer, double, boolean, listing, dictionary, etc.
JSON is very like to YAML and is extremely popular equally well, however, information technology's non possible to add comments in JSON. I use JSON a lot for internal config inside the plan, only not when I want to share the config with other people.
TOML, on the other hand, is similar to INI, but supports more data types and has divers syntax for nested structures. It'south used a lot past Python bundle managements similar pip or poetry. But if the config file has too many nested structures, YAML is a better choice. The following file looks like INI, but every string value has quotes.
Then far, I've explained WHY and WHAT. In the side by side sections, I will show you the HOW.
Option1: YAML/JSON — But read an external file
As usual, we offset from the about basic approach, that is simply creating an external file and reading information technology. Python has dedicated born packages to parse YAML and JSON files. Every bit you see from the lawmaking below, they actually return the aforementioned dict object, so the aspect access volition be the same for both files.
Read
Due to security event, it is recommended to use yaml.safe_load() instead of yaml.load() to avert code injection in the configuration file.
Validation
Both packages will raise a FileNotFoundError for a non-existing file. YAML throws different exceptions for a not-YAML file and an invalid YAML file, while JSON throws JSONDecoderError for both errors.
Option2: Cofigureparser — Python built-in package
From this onwards, I will innovate packages designed for configuration management. Nosotros start with a Python congenital-in packet: Configureparser.
Configureparser is primarily used for reading and writing INI files, but information technology likewise supports dictionary and iterable file object as an input. Each INI file consists of multiple sections where there are multiple key, value pairs. Beneath is an example of accessing the fields.
Read
Configureparser doesn't gauge datatypes in the config file, then every config is stored equally a cord. Only it provides a few methods to catechumen string to the correct datatype. The most interesting one is Boolean type every bit it's able to recognize Boolean values from 'yes'/'no', 'on'/'off', 'truthful'/'false' and 'one'/'0'.
Equally mentioned before, it could too read from a dictionary using read_dict(), or a string using read_string() or an iterable file object using read_file().
Validation
The validation of Configureparser is non as straightforward as YAML and JSON. First, it doesn't raise a FileNotFoundError if the file doesn't exist, simply instead, it raises a KeyError when it tries to access a primal.
Besides, the package "ignores" the fault of indentation. Similar the case below, if you lot have an extra tab or space before "DEBUG", then you would get a wrong value for both ENVIRONMENT and DEBUG.
Still, Configureparser is able to return ParserError for multiple errors (meet the last exam instance). This helps us to solve bug in one shot.
Option3: python-dotenv — Make configurations equally environment variables
Now we move to third-party libraries. So far, I have actually missed one type of configuration files which is .env. The variables inside .env file volition be loaded equally environment variables by python-dotenv and can be accessed by os.getenv.
A .env file basically looks similar this. The default path is the root folder of your project.
Environs=test
DEBUG=truthful
USERNAME=xiaoxu
Password=xiaoxu
HOST=127.0.0.1
PORT=5432 Read
It is extremely easy to use. You lot tin determine whether you lot want to override the existing variable in the surround with parameter override.
Validation
However, python-dotenv doesn't validate the .env file. If you accept a .env file like this, and you want to access DEBUG, you lot will get None as the return without an exception.
# .env
Surroundings=exam
DEBUG # load.py
load_dotenv()
print('DEBUG' in os.environ.keys())
# False
Option4: Dynaconf — Powerful settings configuration for Python
Dynaconf is a very powerful settings configuration for Python that supports multi file formats: yaml, json, ini, toml and python. It tin can automatically load .env file and supports custom validation rules. In a short, it covers pretty much all the functionalities from the previous 3 options and even goes beyond that. For instance, y'all can store an encrypted password and use a custom loader to decrypt the password. Information technology's also nicely integrated with Flask, Django and Pytest. I volition not mention all the functionalities in this article, for more details, please refer to their documentation.
Read
Dynaconf uses .env to find all the settings file and populate settings object with the fields. If 2 settings file have the aforementioned variable, then the value volition be overridden by the latest settings file.
Validation
One of the interesting features to me is the custom validator. As mentioned earlier, Configureparser doesn't validate INI file strictly plenty, but this can be achieved within dynaconf. In this instance, I check whether certain keys be in the file and whether certain primal has the correct value. If y'all read from YAML or TOML file which supports multi datatypes, you can even check if a number is in a certain range.
Integration with Pytest
Another interesting feature is the integration with pytest. The settings for unit testing are normally different from other environments. You can apply FORCE_ENV_FOR_DYNACONF to permit the application read a unlike section in your settings file, or utilise monkeypatch to replace a speific cardinal and value pair in the settings file.
Refresh the config during Runtime
Dynaconf also supports reload() , that cleans and executes all the loaders. This is helpful if you want the application to reload the settings file during runtime. For example, the application should automatically reload the settings when the config file has been opened and modified.
Option5: Hydra- Simplify the development by dynamically creating a hierarchical configuration
The last choice is much more than than simply a file loader. Hydra is a framework developed by Facebook for elegantly configuring complex applications.
Besides reading, writing and validating config files, Hydra as well comes up with a strategy to simplify the management of multi config files, override information technology through command line interface, create a snapshot of each run and etc.
Read
Hither is the basic use of hydra. +APP.NAME means adding a new field in the config, or APP.Proper noun=hydra1.1 to override an existing field.
Validation
Hydra nicely integrate with @dataclass to perform basic validations such as type checking and read-only fields. But it doesn't support __post_init__ method for advanced value checking like described in my previous article.
Config group
Hydra introduces a concept chosen config grouping. The idea is to group configs with the same type and choose ane of them during the execution. For example, you tin can have a group "database" with one config for Postgres, and another one for MySQL.
When information technology gets more than complex, yous might take a layout like this in your program (an case from Hydra documentation)
and you desire to benchmark your application with different combinations of db, schema and ui, then yous can run:
python my_app.py db=postgresql schema=school.yaml More …
Hydra supports parameter sweep with --multirun, that runs multiple jobs at the same with different config files. For instance, for the previous instance, we can run:
python my_app.py schema=warehouse,support,schoolhouse db=mysql,postgresql -m Then you basically start half-dozen jobs simultaneously
[2019-10-01 14:44:xvi,254] - Launching half-dozen jobs locally
[2019-x-01 xiv:44:16,254] - Sweep output dir : multirun/2019-x-01/14-44-16
[2019-10-01 14:44:16,254] - #0 : schema=warehouse db=mysql
[2019-10-01 14:44:16,321] - #i : schema=warehouse db=postgresql
[2019-10-01 fourteen:44:sixteen,390] - #two : schema=support db=mysql
[2019-x-01 14:44:16,458] - #iii : schema=support db=postgresql
[2019-10-01 14:44:16,527] - #4 : schema=school db=mysql
[2019-ten-01 14:44:sixteen,602] - #5 : schema=school db=postgresql Conclusion
In this article, I've talk virtually configuration management in Python in terms of WHY, WHAT and HOW. Depending on the usecase, a complex tool/framework isn't always better than a simple parcel. No matter which one you cull, you should always think nearly its readbility, matainbility and how to spot the error as earily as possible. In fact, config file is just another type of code.
I hope you lot enjoy this article, and experience free to go out your comments below.
Reference
Source: https://towardsdatascience.com/from-novice-to-expert-how-to-write-a-configuration-file-in-python-273e171a8eb3
0 Response to "Simple Web Page to Read Write Config File"
Post a Comment