﻿ The only unbreakable cryptosystem known - the Vernam cipher

# The only unbreakable cryptosystem known - the Vernam cipher

• The only unbreakable cryptosystem known - the Vernam cipher
• One-time pad encryption in Python Breaking Code
• One-Time Pad or vernam Cipher.ppt Key (Cryptography ...
• A Unary Cipher with Advantages over the Vernam Cipher
• Writing the Next Chapter for the Historic One-Time Pad
• One-Time Pad Encryption: Example & Definition Study.com

##### [Ciphers for beginners] Chapter 2: Binary-to-text encodings

In last chapter, we talked about how characters could be encoded using different numeral systems. We'll stay on the topic of encodings a bit longer to talk about binary-to-text encodings. Those algorithms are used by computers to exchange any sort of data using only a restricted set of characters. Nowadays, it is not rare to see them used in ARGs to encrypt messages.
Before we dive further into this topic, note that those encodings can be used to represent any data, not only text. You could use a binary-to-text encoding to represent the data of an image or a soundtrack, for example.

# Base64

The most famous of those encodings is called Base64. If you read the previous chapter, then you might already guess where this is going. Base64 can be considered as a numeral system that uses 64 characters: A to Z, a to z, 0 to 9 and the characters + and /. With each of those is associated an index, using the index table below:
The Base64 index table
To encode using Base64, each character from the plaintext must first be converted to its corresponding ASCII binary code. Then, each group of 6 bits is converted to a Base64 character by using the index table.
Using Base64, the message "ARG" would be encoded as "QVJH".
How Base64 encoding works
Because bytes contain 8 bits each, 3 bytes (3 x 8 = 24 bits) can be encoded as 4 Base64 characters (4 x 6 = 24 bits). For this reason, most Base64 messages have a length that is a multiple of 4. A 65th character, =, is usually used at the end to pad the length of the message if need be.
There also exists a number of variants of Base64, using different characters in place of +, / and =. But don't worry too much about this: ARGs that use Base64 usually choose the most common version of it.

# Base32

Other similar encodings exist, such as Base32, which uses only 32 characters: A to Z and 2 to 7, with again optional = for padding.
The Base32 index table
It works the same way as Base64 but the bits are grouped by 5 this time. 5 bytes of data (5 x 8 = 40 bits) are encoded using 8 Base32 characters (8 x 5 = 40 bits). If the length of the result is not a multiple of 8 already, it is usually padded using the extra character =.
Using Base64, the message "ARG" would be encoded as "IFJEO===".
How Base32 encoding works
Again, just like for Base64, note that there exists different variants of Base32, using different sets of characters.

# Base58, Base91

Still in the same vein, you might on rare occasions encounter less used text-to-binary encodings.
For example, Base58 uses the same characters as Base64 but omits 0, O, I, l, + and /.
Base91 uses almost all printable ASCII characters except the space, the dash, the backslash and the apostrophe.
They work a bit differently than Base64 but considering they are rarely used, there’s probably no need to dive into their precise behavior. Just try to remember they exist, so that you’re ready once you encounter them!

# Base36

Base36 is yet another binary-to-text encoding, this time using the characters A to Z and 0 to 9. It can in fact really be seen as a normal numeral system, where you count from 0 to 9, then goes on with A (10 in decimal) up to Z (35 in decimal). To write 36 in Base36, you would need to add another digit, leading to 10. And so on...
An interesting fact about this encoding is that it can be used to hide an alphanumeric message by actually decoding from Base36. Indeed, any alphanumeric message (with no space) is a valid Base36 number! For example, ARG in Base36 is 13948 in decimal.
Base36 to decimal conversion

# Ascii85

Finally, let's also mention the encoding called Ascii85, sometimes referred to as Base85. The original version of it uses the characters 0 to 9, A to Z, a to u, !"#\$%'()*+,-./:;<=>[email protected][\]^_, the backtick ​` and sometimes y and z.
Another version, created by Adobe, uses the same alphabet but added <~ and ~> delimiters around the code.
And last but not least, it exists a version called Z85, which uses the characters 0 to 9, a to z, A to Z, and .-:+=^!/*?&<>()[]{}@%\$#.
The algorithm for Ascii85 is a bit more complex than the previous ones but explaining it is probably not needed. Don’t worry though, you can easily find online tools to do the decoding for you!

## Quickly identify binary-to-text encoding

Phew, what a mess, right? Let's sum this up real quick.
When you encounter a coded message that contains a bunch of characters without space, ask yourself the following questions.
Does it have one or more = at the end? This is characteristic of Base32 and Base64.
Does it contain both uppercase and lowercase letters? Try Base64 or, less likely, Base58.
Does it contain only uppercase letters and digits but you can't find any 0, 1, 8, or 9? You should probably try decoding it with Base32.
Does it contain weird symbol characters? Looks like Base91 or a variant of Ascii85.
Does it start with <~ and/or ends with ~>? This is most likely the Adobe variant of Ascii85.

## Resources

You can find online tools to decode all the binary-to-text encodings listed in this chapter. Here is a non-exhaustive list of them:
As you might have guessed, any base could actually be used to encode binary data, given the proper index table or algorithm. This chapter focused on the most popular ones, that you might encounter in ARGs.
See you in the next chapter!

So I'm gonna dump some info I have gathered one more time. Nothing too cool. I hope you find it useful and maybe this helps to decrypt some of the messages! For now, it's all been failure on my end!

Decrypted posts:
F04_nod.redd -> Even though it's decrypted, I'm not sure. It may be a key for something

Some nice resources:
> Cryptography
> ARGs/internet mysteries/creepypastas

> Steganography

Paige12_2002
- Analysis of a plain english text encoded in base32 against the long message:
`>>>>>>>>>>>>>>>>>>PLAIN ENGLISH ENCODED LEN -> 653 IC -> 0.0369 ###################################### D -> 38 (12.14) 3 -> 38 (12.14) H -> 36 (11.50) X -> 32 (10.22) R -> 30 (9.58) P -> 29 (9.27) 8 -> 29 (9.27) M -> 27 (8.63) F -> 27 (8.63) 9 -> 27 (8.63) ###################################### W7 -> 15 (16.48) D3 -> 11 (12.09) PM -> 11 (12.09) 3R -> 11 (12.09) BX -> 8 (8.79) C8 -> 7 (7.69) X3 -> 7 (7.69) RA -> 7 (7.69) HK -> 7 (7.69) E3 -> 7 (7.69) ###################################### 3RA -> 6 (13.33) X3R -> 5 (11.11) 3RK -> 5 (11.11) 9BX -> 5 (11.11) KM3 -> 4 (8.89) 3DE -> 4 (8.89) W7Z -> 4 (8.89) 8W7 -> 4 (8.89) 7ZH -> 4 (8.89) XE3 -> 4 (8.89) <<<<<<<<<<<<<<<<< - The first message has only 32 different characters (23456789ABCDEFGHJKLMNPQRSTUVWXYZ) in a message that is 695 chars long which suggest some sort of Base32 encoding - The second message has 13 words of 13 letters with a charset of 36 (if we count the space) different characters. Some of the characters here are not present in the first message (012345689ABCDEFGHIJKLMNOPQRSTUVWXYZ) - Could this be a matrix for a Hill Cipher? -> https://en.wikipedia.org/wiki/Hill_cipher - "To help one is to help all" -> may come from the law of one by ra > https://truthearth.org/2016/07/01/divide-and-conquer-a-cabal-tool-as-old-as-time/ > https://www.lawofone.info/ + How to attack this: - One way the first message could be encrypted is by using a custom base32 alphabet Steppos:  Set a randomly sorted base32 alphabet Decrypt the encrypted message using it Check the fitness of the result Modify the alphabet  Seems straight forward. You can't check all alphabet permutations because 32! = 263130836933693530167218012160000000 What do then? - Define transformations of the alphabet like swapping elements, sliding pieces of the alphabet, shuffle chunks,... - Swap failing characters in the alphabet (those that decrypt to non-printable characters) - Define a fitness function that depends on the english frequencies of bigrams, trigrams or quadgrams. Or maybe one based on the printability of the output - If we assume that most of the characters are in the range A-Za-z then we can set a rule: Let us analyze the first character of the string: V V can be any number from 0 to 31... or can it? See, if we assume that the first character is a letter (which may not be the case if the original text is shuffled before the encryption), and also a capital letter (maybe it's the "T" from "To help ..."), then we have a couple of restrictions on what V can be. Ascii uppercase letters have values ranging from 0x41 (01000001) to 0x5a (01011010) so V's value must be 01---. The catch here is the space character (' ' 0x20 00100000) which doesn't start with 01 and can be frequent in the text. Other punctuation symbols have similar issues. ​ past.meeting/html - My best guess here is that this seemingly random chunk of html is to be hashed and a key generated from that, or used as-is as some sort of encryption key. - Source of the chunk -> https://web.archive.org/web/20060402193231/http://dizzygoo.com/ ​ Scout_Cub - The easy solution that seems to be to easy to be a solution -> THEJUNGLEBOOK - Notice it's 13 characters long (can this be used with other 13-char long strings that are present throughout the subreddit?) - Tried to use this as an OTP key for the water_swift with no luck using either the numbers or the letters (I think I did this, but give it a try just in case I didn't do it or did it wrong) ​ barnside_downside - A mime with a broom. It might not be an original image but, if it isn't, I haven't been able to locate the original. - Outguess returns that no bits are available when attempting to decrypt. This seems weird, but I don't know if it means something - Most of the pictures seem to be related with other ARGs/internet mysteries/creepypastas - On the side of the right shack, we can see on the roof a strange drawing and XY + How to attack this: - It might have some hidden info so you can check steganography software like outguess and attempt to recover the hidden information bruteforcing the key with a list of words. Since we don't know wether there is information hidden (or even if reddit compresses the images when uploading them, which would kill any chance of hiding stuff in it), this might lead to nothing. ​ squared_squared - Charset (57): 03456789BCDEFGHIJKLMNOPQRSTUVWXZabcdefghijklmnopqrsuvwxyz - 13 characters per string, 13 strings - This could be a table of keys. + How to attack? - No idea. So start with the basics: - Frequency analysis:  ('0', 9), ('r', 8), ('b', 7), ('q', 6), ('m', 5), ('H', 5), ('J', 5), ('9', 5), ('8', 4), ('g', 4), ('a', 4), ('N', 4), ('x', 4), ('F', 4), ('V', 4), ('h', 3), ('j', 3), ('E', 3), ('S', 3), ('e', 3), ('O', 3), ('v', 3), ('C', 3), ('f', 3), ('z', 3), ('7', 3), ('n', 3), ('o', 3), ('X', 3), ('W', 3), ('d', 2), ('K', 2), ('Z', 2), ('k', 2), ('B', 2), ('G', 2), ('s', 2), ('y', 2), ('Q', 2), ('c', 2), ('T', 2), ('5', 2), ('p', 2), ('i', 2), ('R', 2), ('l', 2), ('3', 2), ('I', 2), ('L', 2), ('P', 1), ('D', 1), ('U', 1), ('w', 1), ('4', 1), ('6', 1), ('u', 1), ('M', 1)  - A similar analysis as with the Paige12 post can be done. target_manners_quiz - Charset (31): 01245679ABCDEFGIJKMNOPQRSUVWXYZ - The fact that it's 31 different characters may come from the fact that the message is not very long or it may be that there are only 31 characters in the alphabet. - It's likely that the top message uses the same alphabet - In the top message not all words are the same length (13 - 6 - 12 - 6 - 13) ​ random_chance/fate - Another chunk of html. - A similar one -> https://codepen.io/Keyy/pen/GmdWwX.html - Might be an old implementation of some sort of MM Chat -> https://www.cvedetails.com/vulnerability-list/vendor_id-8299/year-2008/Mm-Chat.html - 859 chars long ​ water_swift - 13 hexadecimal character - This could be a One Time Pad (OTP). In this case what you do is you take a string that is also 13 chars long and xor each character with a 13 char long key. To decrypt, xor the encrypted result with the key. + How to attack: - Take all 13 char long strings and xor them against this. - The problem with OTP is that unless you know the key, you can make the text say anything you want by decrypting it with the right key: >> Let's assume I want this to be THEJUNGLEBOOK. What I need to do is xor each character of the string with the encrypted message. That gives me c1 db ea 0c c3 71 22 83 a2 7d 61 79 4d. I can now use this to xor the encrypted message and get THEJUNGLEBOOK as plaintext ​ archive-hiking - A link to a section of oocities and a series of numbers and words. Will do a bot to check them sites! >> BOT OUTPUT: Found omega in http://www.oocities.org/yosemite/3949/ Found 1313 in http://www.oocities.org/yosemite/4321/ Found allag in http://www.oocities.org/yosemite/5934/ Found omega in http://www.oocities.org/yosemite/6152/ Found omega in http://www.oocities.org/yosemite/6195/ Found 1313 in http://www.oocities.org/yosemite/8423/ Found omega in http://www.oocities.org/yosemite/8878/ Found 1313 in http://www.oocities.org/yosemite/9185/ Found 1313 in http://www.oocities.org/yosemite/9195/  - This was found crawling only the main index. The bot may need to go deeper underground! - The words are 13 13 omega allag weinstein challa g57 - g57 may be another medical code -> https://icd.codes/icd10cm/G57 ​ 8clubs_1anni - Charset (32): 23456789ABCDEFGHJKLMNPQRSTUVWXYZ - 20 8-letter strings - It may use the same algorithm as the first post ​ rich_s - Another chunk of html (1507 chars) - Seems to come from facebook somehow. Similar code -> https://codepen.io/dev2309/pen/XpaBxo.html 20feb04stockwood.cplus - A c program which outputs something like this: 1313 >> 20 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210  - >> indicates the input I gave it - This could be used to sort some of the characters in the encrypted messages. - The encryption could be just writing the characters in order and then take them out in columns. For example: - We want to encrypt ATTACKATDAWN - We lay the text according to this pyramid  A TT ACK ATDA WN  - Extract the text by columns -> ATAAWTCTWKDA - To decrypt, just lay the text in columns and enjoy. Combine this with some sort of substitution to reduce your sanity levels! - A python implementation: def main(): n, i, c, a = (1, 1, 1, 1) print('1313') # Will break if a string is input n = int(input()) # Since python's range goes [1,n) we need to increase n by 1 # we need to add one to the top limit for i in range(1, n+1): for c in range(1, i+1): print('%d ' % a, end='') a += 1 print() return 0  missing_param - Here starts the shit. The encrypted text (if it is an encrypted message), not only plays with the different characters, but it also adds formatting to the equation. Is this important? Can I ignore italics and other such artifacts? Probably not. Probably not... ​ passable_forest - This looks like substitution + transposition. Are the spaces moved around to? If not, how many combinations that make sense of a one-letter - two-letter pair are there in english? - Give it a try with the program of the stockwood post - Check them candidate algorithms: > Vigenere > Autokey > Beaufort > Running key > Hill cipher > ADFGVX cipher > Playfair cipher > Moar -> https://en.wikipedia.org/wiki/Category:Classical_ciphers ​ homage_com_dos_f04 - This one seems to be a variation of the f04cb algorithm. - If we arrange the characters into columns every 3 bytes we get:  3d 41 74 3c 42 73 38 43 73 3a 43 71 36 40 72 36 45 76 3c 40 76 3d 47 78 35 a9 75 38 41 71 39 a3 74 35 41 71 39 a5 78 35 a9 77 35 42 77 39 a5 72 3d 42 76 3a 44 5a 39 a6 73 36 40 76 3a 48 74 36 40 74 37  - All characters in the left column start with 3 (0011) - Almost all characters in the middle column start with 4 (0100) - Almost all characters in the right column start with 7 (0111) - The fact that it's almost all points to a operation done to the characters (like a xor) and not to half-bytes just being added in between half-bytes - It is impossible to represent more than 16 characters with half a byte. This means that, in case the xor operation was the last one done, you can't know in advance where the positions of the 0s and 1s in the first half of the byte are going to be. You could have some sort of one time pad so that it produces this result, but this option seems unlikely - Those characters that don't follow the pattern could correspond with special characters (like \r, \n, \t,...) - Since 3 seems to have some sort of significance, it could be that the operations are done to groups of 3 bits (or 6, 13, pick a number!) ​ welcome to our new home - Just a text. 32 characters long if we take the spaces into account ​ anthem_crab.tree - Not your typical Lorem Ipsum. It starts with the standard "Lorem ipsum" but then changes. - All the words seem to be in the post seem to be in the original post. Maybe this is a clue for the code. Maybe the text can be translated into numbers according to the place of the word in the original text (first occurrence). Then, maybe those numbers can be used for something. Maybe. - The tree in the title may be a clue to use the stockwood program ​ hollow_web.jpg - As with all images, check with outguess or similar tools - Original image -> https://archive.4plebs.org/x/thread/14736756/ ​ verse.txt.doc - The text seems to be some dummy text - You can check this by searching for example "Gi tractare ut ex concilia" in google - Where this text comes from is unknown - A longer version -> https://koouma2.blogspot.com/2019/01/de-iste-tunc-esse-illa-ii-actu-idem.html ​ start.search.ggd - Another reference to oocities - The text is from Kafka's Metamorphosis -> https://dailylit.com/read/175-the-metamorphosis - It also seems to be used as dummy text for html templates ​ creations_puzzle.jpg - Crossword picture. Could this be used as a template for letters in some of the previous text? - Following are the clues with the length of each word in the crosswords ACROSS 3 The OG (6) 5 Lake (4) 10 Brookfeld (3) 12 The first coming (14) 14 The Great SF64 (4) 15 Moose (8) 16 Prefix and begin (3) 17 Justic (4) DOWN 1 Ema (5) 2 VolumeX (9) 4 Fasttrack (3) 6 Meet You There (5) 7 Frame (10) 8 The Lost (7 or 6) 9 Pursuit (7 or 6) 11 Microphone (4) 12 IceRen (5) 13 Jacket (4) - Please check that the words have been transcribed correctly ​ EAW - CYFI? - Looks like one of those transposition + substition ciphers I heard so much about... - That EAW, has it something to do with the Ema of the creations_puzzle.jpg? + How to attack - Get frequencies of letters and bigrams - Get the Index of Coincidence - Depending on what comes out cry or attempt something different - First, try key sizes of 6 and 13 as they seem to be important numbers - Try to guess some of the words that can be there. The longer the better! - When fail, crouch into fetal position and keep crying. jan_24_2015.cal - A program?  // # # # # # # # y=0 // # # . . . # # y=1 // # # . . . . # y=2 // # . . . . . # y=3 // # # . . . . # y=4 // # # . . . # # y=5 // # # # # # # # y=6 x2 = x - 1 + ((y + FirstShift) % 2); x3 = x + ((y + FirstShift) % 2);  - If it is an % represents modulo, then the second part can only be 0 or 1 - x3 = x2 + 1 - There is also a list of numbers and a sentence - Is the reference a shady reference to cicada? - Is it a reference to one of the spinoffs of cicada? ​ wonder_year_s4_20 - Charset (24): _'ABCDEFGHIKLMNOPRSTUVWY (the _ represents the space) - It's 143 characters long. 143 = 13*11 - The IC matches that of english so it could be that only transposition has been used - Individual frequencies also match those of english (more or less, but good enough for such a small text) - At least it's a double transposition (maybe more) - There are 24 spaces which suggest that the sentence is 25 words long - The presence of ' indicates that there is a n't or 's (are there more possibilities?) - We don't know the key size. - Key lengths -> first I'll try 13 13, 6 13, 13 6, 4 20, 20 4 and see what happens - Maybe the key is in one of the previous messages with long strings or even the jungle book one + How to attack? - Assuming this is a double transposition, follow this -> https://www.uni-kassel.de/upress/online/OpenAccess/978-3-7376-0458-1.OpenAccess.pdf chapter 5.3.1 (page 68) - If it's more than double transposition, I think the same attack vector still holds - Check also William Friedman's literature on the subject ​ bright_lights.doc - Charset(32): 23456789ABCDEFGHJKLMNPQRSTUVWXYZ - Here is the reference to 1976. It could be a reference to the paper by Diffie and Hellman - All strings are 8 chars long except for the first and the last (6) ​ pix_strand - No idea. Has a comment that looks like a perl script. Haven't tried to run it (it would need a couple of files) ​ ​ I'm going to add a small explanation on baseN numbers What is baseN? A base32, base64, base10,... is just the number of different characters you use to represent a number. For example  base10 base2 base16 12 1100 0xc  ​ This can be interpreted as:  12 -> 1 * 10^1 + 2 * 10^0 1100 -> 1 * 2^3 + 1 * 2^2 + 0 * 2^1 + 0 * 2^0 0xc -> c * 16 ^ 0  In the case of base 16 we need more characters than just the numbers from 0-9 so we use a-f for the 10-15 range. In this example c = 12. This is the basic idea. However when you check the standard base64 implementations you can see that the encoded string has sometimes padding characters (=). This is a consequence of how bytes are encoded into base64 to optimize the performance of the algorithm. In the standard case, each of the characters of the base64 alphabet (https://en.wikipedia.org/wiki/Base64#Base64_table) from 0 (binary 000000) to 63 (binary 111111). So when you encode a string like MESSAGE to base64 first you transform it into bits:  M E S S A G E 01001101 01000101 01010011 01010011 01000001 01000111 01000101  Then, group them into 6bit numbers:  >> 010011 010100 010101 010011 010100 110100 000101 000111 010001 01  We need to add 4 zeros to complete the 6 bits groups. To indicate this we will add 2 = chars to the end of the string  >> 010011 010100 010101 010011 010100 110100 000101 000111 010001 010000  If we encode this according to the base64 value table we get:  010011 010100 010101 010011 010100 110100 000101 000111 010001 010000 T U V T U 0 F H R Q ==  ​ `

# Caution:

## This is a long doc covering every changes in python 3.6.If you do not know anything about the commands do not read this!

Manuel and other things of Python 3.6

# Python 3.6

## What’s New In Python 3.6

Editors: Elvis Pranskevichus [email protected], Yury Selivanov [email protected] This article explains the new features in Python 3.6, compared to 3.5. Python 3.6 was released on December 23, 2016.

## Summary – Release highlights

New syntax features:
• PEP 498, formatted string literals.
• PEP 515, underscores in numeric literals.
• PEP 526, syntax for variable annotations.
• PEP 525, asynchronous generators.
• PEP 530: asynchronous comprehensions.
New library modules:
• secrets: PEP 506 – Adding A Secrets Module To The Standard Library.
CPython implementation improvements:
• The dict type has been reimplemented to use a more compact representation based on a proposal by Raymond Hettinger and similar to the PyPy dict implementation. This resulted in dictionaries using 20% to 25% less memory when compared to Python 3.5.
• Customization of class creation has been simplified with the new protocol.
• The class attribute definition order is now preserved.
• The order of elements in **kwargs now corresponds to the order in which keyword arguments were passed to the function.
• DTrace and SystemTap probing support has been added.
• The new PYTHONMALLOC environment variable can now be used to debug the interpreter memory allocation and access errors.
Significant improvements in the standard library:
• The asyncio module has received new features, significant usability and performance improvements, and a fair amount of bug fixes. Starting with Python 3.6 the asyncio module is no longer provisional and its API is considered stable.
• A new file system path protocol has been implemented to support path-like objects. All standard library functions operating on paths have been updated to work with the new protocol.
• The datetime module has gained support for Local Time Disambiguation.
• The typing module received a number of improvements.
• The tracemalloc module has been significantly reworked and is now used to provide better output for ResourceWarning as well as provide better diagnostics for memory allocation errors. See the PYTHONMALLOC section for more information.
Security improvements:
• The new secrets module has been added to simplify the generation of cryptographically strong pseudo-random numbers suitable for managing secrets such as account authentication, tokens, and similar.
*On Linux, os.urandom() now blocks until the system urandom entropy pool is initialized to increase the security. See the PEP 524 for the rationale.
• The hashlib and ssl modules now support OpenSSL 1.1.0.
• The default settings and feature set of the ssl module have been improved.
• The hashlib module received support for the BLAKE2, SHA-3 and SHAKE hash algorithms and the scrypt() key derivation function.
Windows improvements:
• PEP 528 and PEP 529, Windows filesystem and console encoding changed to UTF-8. The py.exe launcher, when used interactively, no longer prefers Python 2 over Python 3 when the user doesn’t specify a version (via command line arguments or a config file). Handling of shebang lines remains unchanged - “python” refers to Python 2 in that case.
• python.exe and pythonw.exe have been marked as long-path aware, which means that the 260 character path limit may no longer apply. See removing the MAX_PATH limitation for details. A ._pth file can be added to force isolated mode and fully specify all search paths to avoid registry and environment lookup.

# New Features

## PEP 498: Formatted string literals

PEP 498 introduces a new kind of string literals: f-strings, or formatted string literals. Formatted string literals are prefixed with 'f' and are similar to the format strings accepted by str.format(). They contain replacement fields surrounded by curly braces. The replacement fields are expressions, which are evaluated at run time, and then formatted using the format() protocol:
name = "Fred">>> f"He said his name is {name}."'He said his name is Fred.'>>>
width = 10>>> precision = 4>>> value = decimal.Decimal("12.34567")>>> f"result:
{value:{width}.{precision}}" # nested fields'result: 12.35'

# PEP 526: Syntax for variable annotations

PEP 484 introduced the standard for type annotations of function parameters, a.k.a. type hints. This PEP adds syntax to Python for annotating the types of variables including class variables and instance variables:
primes: List[int] = []
captain: str # Note: no initial value!
class Starship:
`stats: Dict[str, int] = {} `
Just as for function annotations, the Python interpreter does not attach any particular meaning to variable annotations and only stores them in the annotations attribute of a class or module. In contrast to variable declarations in statically typed languages, the goal of annotation syntax is to provide an easy way to specify structured type metadata for third party tools and libraries via the abstract syntax tree and the annotations attribute.
Tools that use or will use the new syntax: mypy, pytype, PyCharm, etc.

# PEP 515: Underscores in Numeric Literals

PEP 515 adds the ability to use underscores in numeric literals for improved readability. For example:
1_000_000_000_000_0001000000000000000>>>
0x_FF_FF_FF_FF4294967295
Single underscores are allowed between digits and after any base specifier. Leading, trailing, or multiple underscores in a row are not allowed.
The string formatting language also now has support for the '_' option to signal the use of an underscore for a thousands separator for floating point presentation types and for integer presentation type 'd'. For integer presentation types 'b', 'o', 'x', and 'X', underscores will be inserted every 4 digits:
'{:_}'.format(1000000)'1_000_000'>>> '{:_x}'.format(0xFFFFFFFF)'ffff_ffff'

# PEP 525: Asynchronous Generators

PEP 492 introduced support for native coroutines and async / await syntax to Python 3.5. A notable limitation of the Python 3.5 implementation is that it was not possible to use await and yield in the same function body. In Python 3.6 this restriction has been lifted, making it possible to define asynchronous generators:
async def ticker(delay, to):
`"""Yield numbers from 0 to *to* every *delay* seconds.""" for i in range(to): yield i await asyncio.sleep(delay) `
The new syntax allows for faster and more concise code.

# PEP 530: Asynchronous Comprehensions

PEP 530 adds support for using async for in list, set, dict comprehensions and generator expressions:
result = [i async for i in aiter() if i % 2]
Additionally, await expressions are supported in all kinds of comprehensions:
result = [await fun() for fun in funcs if await condition()]

# PEP 487: Simpler customization of class creation

It is now possible to customize subclass creation without using a metaclass. The new init_subclass classmethod will be called on the base class whenever a new subclass is created:
class PluginBase:
`subclasses = [] def __init_subclass__(cls, **kwargs): super().__init_subclass__(**kwargs) cls.subclasses.append(cls) `
class Plugin1(PluginBase):
`pass `
class Plugin2(PluginBase):
`pass `
In order to allow zero-argument super() calls to work correctly from init_subclass() implementations, custom metaclasses must ensure that the new classcell namespace entry is propagated to type.new (as described in Creating the class object).

# PEP 487: Descriptor Protocol Enhancements

PEP 487 extends the descriptor protocol to include the new optional set_name() method. Whenever a new class is defined, the new method will be called on all descriptors included in the definition, providing them with a reference to the class being defined and the name given to the descriptor within the class namespace. In other words, instances of descriptors can now know the attribute name of the descriptor in the owner class:
class IntField:
`def __get__(self, instance, owner): return instance.__dict__[self.name] def __set__(self, instance, value): if not isinstance(value, int): raise ValueError(f'expecting integer in {self.name}') instance.__dict__[self.name] = value # this is the new initializer: def __set_name__(self, owner, name): self.name = name `
class Model:
`int_field = IntField() `

# PEP 519: Adding a file system path protocol

File system paths have historically been represented as str or bytes objects. This has led to people who write code which operate on file system paths to assume that such objects are only one of those two types (an int representing a file descriptor does not count as that is not a file path). Unfortunately that assumption prevents alternative object representations of file system paths like pathlib from working with pre-existing code, including Python’s standard library. To fix this situation, a new interface represented by os.PathLike has been defined. By implementing the fspath() method, an object signals that it represents a path. An object can then provide a low-level representation of a file system path as a str or bytes object. This means an object is considered path-like if it implements os.PathLike or is a str or bytes object which represents a file system path. Code can use os.fspath(), os.fsdecode(), or os.fsencode() to explicitly get a str and/or bytes representation of a path-like object. The built-in open() function has been updated to accept os.PathLike objects, as have all relevant functions in the os and os.path modules, and most other functions and classes in the standard library. The os.DirEntry class and relevant classes in pathlib have also been updated to implement os.PathLike. The hope is that updating the fundamental functions for operating on file system paths will lead to third-party code to implicitly support all path-like objects without any code changes, or at least very minimal ones (e.g. calling os.fspath() at the beginning of code before operating on a path-like object). Here are some examples of how the new interface allows for pathlib.Path to be used more easily and transparently with pre-existing code:
import pathlib>>> with open(pathlib.Path("README")) as f:... contents
os.path.splitext(pathlib.Path("some_file.txt"))('some_file', '.txt')>>>
os.path.join("/a/b", pathlib.Path("c"))'/a/b/c'>>> import os>>>
os.fspath(pathlib.Path("some_file.txt"))'some_file.txt'

# PEP 495: Local Time Disambiguation

In most world locations, there have been and will be times when local clocks are moved back. In those times, intervals are introduced in which local clocks show the same time twice in the same day. In these situations, the information displayed on a local clock (or stored in a Python datetime instance) is insufficient to identify a particular moment in time. PEP 495 adds the new fold attribute to instances of datetime.datetime and datetime.time classes to differentiate between two moments in time for which local times are the same:
u0 = datetime(2016, 11, 6, 4, tzinfo=timezone.utc)>>> for i in range(4):...
u = u0 + i*HOUR... t = u.astimezone(Eastern)... print(u.time(), 'UTC =',
t.time(), t.tzname(), t.fold)...04:00:00 UTC = 00:00:00 EDT 005:00:00 UTC = 01:00:00
EDT 006:00:00 UTC = 01:00:00 EST 107:00:00 UTC = 02:00:00 EST 0
The values of the fold attribute have the value 0 for all instances except those that represent the second (chronologically) moment in time in an ambiguous case.

# PEP 529: Change Windows filesystem encoding to UTF-8

Representing filesystem paths is best performed with str (Unicode) rather than bytes. However, there are some situations where using bytes is sufficient and correct. Prior to Python 3.6, data loss could result when using bytes paths on Windows. With this change, using bytes to represent paths is now supported on Windows, provided those bytes are encoded with the encoding returned by sys.getfilesystemencoding(), which now defaults to 'utf-8'. Applications that do not use str to represent paths should use os.fsencode() and os.fsdecode() to ensure their bytes are correctly encoded. To revert to the previous behaviour, set PYTHONLEGACYWINDOWSFSENCODING or call sys._enablelegacywindowsfsencoding().

# PEP 528: Change Windows console encoding to UTF-8

The default console on Windows will now accept all Unicode characters and provide correctly read str objects to Python code. sys.stdin, sys.stdout and sys.stderr now default to utf-8 encoding. This change only applies when using an interactive console, and not when redirecting files or pipes. To revert to the previous behaviour for interactive console use, set PYTHONLEGACYWINDOWSSTDIO.

# PEP 520: Preserving Class Attribute Definition Order

Attributes in a class definition body have a natural ordering: the same order in which the names appear in the source. This order is now preserved in the new class’s dict attribute. Also, the effective default class execution namespace (returned from type.prepare()) is now an insertion-order-preserving mapping.

# PEP 468: Preserving Keyword Argument Order

**kwargs in a function signature is now guaranteed to be an insertion-order-preserving mapping.

# New dict implementation

The dict type now uses a “compact” representation based on a proposal by Raymond Hettinger which was first implemented by PyPy. The memory usage of the new dict() is between 20% and 25% smaller compared to Python 3.5. The order-preserving aspect of this new implementation is considered an implementation detail and should not be relied upon (this may change in the future, but it is desired to have this new dict implementation in the language for a few releases before changing the language spec to mandate order-preserving semantics for all current and future Python implementations; this also helps preserve backwards-compatibility with older versions of the language where random iteration order is still in effect, e.g. Python 3.5). (Contributed by INADA Naoki in bpo-27350. Idea originally suggested by Raymond Hettinger.) PEP 523: Adding a frame evaluation API to CPython While Python provides extensive support to customize how code executes, one place it has not done so is in the evaluation of frame objects. If you wanted some way to intercept frame evaluation in Python there really wasn’t any way without directly manipulating function pointers for defined functions. PEP 523 changes this by providing an API to make frame evaluation pluggable at the C level. This will allow for tools such as debuggers and JITs to intercept frame evaluation before the execution of Python code begins. This enables the use of alternative evaluation implementations for Python code, tracking frame evaluation, etc. This API is not part of the limited C API and is marked as private to signal that usage of this API is expected to be limited and only applicable to very select, low-level use-cases. Semantics of the API will change with Python as necessary.

# PYTHONMALLOC environment variable

The new PYTHONMALLOC environment variable allows setting the Python memory allocators and installing debug hooks. It is now possible to install debug hooks on Python memory allocators on Python compiled in release mode using PYTHONMALLOC=debug. Effects of debug hooks: Newly allocated memory is filled with the byte 0xCB Freed memory is filled with the byte 0xDB Detect violations of the Python memory allocator API. For example, PyObject_Free() called on a memory block allocated by PyMem_Malloc(). Detect writes before the start of a buffer (buffer underflows) Detect writes after the end of a buffer (buffer overflows) Check that the GIL is held when allocator functions of PYMEM_DOMAIN_OBJ (ex: PyObject_Malloc()) and PYMEM_DOMAIN_MEM (ex: PyMem_Malloc()) domains are called. Checking if the GIL is held is also a new feature of Python 3.6. See the PyMem_SetupDebugHooks() function for debug hooks on Python memory allocators. It is now also possible to force the usage of the malloc() allocator of the C library for all Python memory allocations using PYTHONMALLOC=malloc. This is helpful when using external memory debuggers like Valgrind on a Python compiled in release mode. On error, the debug hooks on Python memory allocators now use the tracemalloc module to get the traceback where a memory block was allocated. Example of fatal error on buffer overflow using python3.6 -X tracemalloc=5 (store 5 frames in traces):
Debug memory block at address p=0x7fbcd41666f8: API 'o'
`4 bytes originally requested The 7 pad bytes at p-7 are FORBIDDENBYTE, as expected. The 8 pad bytes at tail=0x7fbcd41666fc are not all FORBIDDENBYTE (0xfb): at tail+0: 0x02 *** OUCH at tail+1: 0xfb at tail+2: 0xfb at tail+3: 0xfb at tail+4: 0xfb at tail+5: 0xfb at tail+6: 0xfb at tail+7: 0xfb The block was made by call #1233329 to debug malloc/realloc. Data at p: 1a 2b 30 00 `
Memory block allocated at (most recent call first):
File "test/test_bytes.py", line 323
File "unittest/case.py", line 600
File "unittest/case.py", line 648
File "unittest/suite.py", line 122
File "unittest/suite.py", line 84
Current thread 0x00007fbcdbd32700 (most recent call first):
File "test/test_bytes.py", line 323 in test_hex
File "unittest/case.py", line 600 in run
File "unittest/case.py", line 648 in call
File "unittest/suite.py", line 122 in run
File "unittest/suite.py", line 84 in call
File "unittest/suite.py", line 122 in run
File "unittest/suite.py", line 84 in call
...
(Contributed by Victor Stinner in bpo-26516 and bpo-26564.)

# DTrace and SystemTap probing support

Python can now be built --with-dtrace which enables static markers for the following events in the interpreter: * function call/return * garbage collection started/finished * Line of code executed.
This can be used to instrument running interpreters in production, without the need to recompile specific debug builds or providing application-specific profiling/debugging code. More details in Instrumenting CPython with DTrace and SystemTap.
The current implementation is tested on Linux and macOS. Additional markers may be added in the future.
(Contributed by Łukasz Langa in bpo-21590, based on patches by Jesús Cea Avión, David Malcolm, and Nikhil Benesch.)

# Other Language Changes

Some smaller changes made to the core Python language are: A global or nonlocal statement must now textually appear before the first use of the affected name in the same scope. Previously this was a SyntaxWarning.
It is now possible to set a special method to None to indicate that the corresponding operation is not available. For example, if a class sets iter() to None, the class is not iterable. (Contributed by Andrew Barnert and Ivan Levkivskyi in bpo-25958.)
Long sequences of repeated traceback lines are now abbreviated as "[Previous line repeated {count} more times]" (see traceback for an example). (Contributed by Emanuel Barry in bpo-26823.)
Import now raises the new exception ModuleNotFoundError (subclass of ImportError) when it cannot find a module. Code that currently checks for ImportError (in try-except) will still work. (Contributed by Eric Snow in bpo-15767.)
Class methods relying on zero-argument super() will now work correctly when called from metaclass methods during class creation. (Contributed by Martin Teichmann in bpo-23722.)

# New Modules

## secrets

The main purpose of the new secrets module is to provide an obvious way to reliably generate cryptographically strong pseudo-random values suitable for managing secrets, such as account authentication, tokens, and similar. Warning Note that the pseudo-random generators in the random module should NOT be used for security purposes. Use secrets on Python 3.6+ and os.urandom() on Python 3.5 and earlier.

# Improved Modules

## array

Exhausted iterators of array.array will now stay exhausted even if the iterated array is extended. This is consistent with the behavior of other mutable sequences. Contributed by Serhiy Storchaka in bpo-26492.

## ast

The new ast.Constant AST node has been added. It can be used by external AST optimizers for the purposes of constant folding. Contributed by Victor Stinner in bpo-26146.

## binascii

The b2a_base64() function now accepts an optional newline keyword argument to control whether the newline character is appended to the return value. (Contributed by Victor Stinner in bpo-25357.)

## cmath

The new cmath.tau (τ) constant has been added. (Contributed by Lisa Roach in bpo-12345, see PEP 628 for details.)
New constants: cmath.inf and cmath.nan to match math.inf and math.nan, and also cmath.infj and cmath.nanj to match the format used by complex repr. (Contributed by Mark Dickinson in bpo-23229.)

## collections

The new Collection abstract base class has been added to represent sized iterable container classes. (Contributed by Ivan Levkivskyi, docs by Neil Girdhar in bpo-27598.)
The new Reversible abstract base class represents iterable classes that also provide the reversed() method. (Contributed by Ivan Levkivskyi in bpo-25987.)
The new AsyncGenerator abstract base class represents asynchronous generators. (Contributed by Yury Selivanov in bpo-28720.)
The namedtuple() function now accepts an optional keyword argument module, which, when specified, is used for the module attribute of the returned named tuple class. (Contributed by Raymond Hettinger in bpo-17941.)
The verbose and rename arguments for namedtuple() are now keyword-only. (Contributed by Raymond Hettinger in bpo-25628.)
Recursive collections.deque instances can now be pickled. (Contributed by Serhiy Storchaka in bpo-26482.)

## concurrent.futures

The ThreadPoolExecutor class constructor now accepts an optional thread_name_prefix argument to make it possible to customize the names of the threads created by the pool. (Contributed by Gregory P. Smith in bpo-27664.)

## contextlib

The contextlib.AbstractContextManager class has been added to provide an abstract base class for context managers. It provides a sensible default implementation for enter() which returns self and leaves exit() an abstract method. A matching class has been added to the typing module as typing.ContextManager. (Contributed by Brett Cannon in bpo-25609.)

## datetime

The datetime and time classes have the new fold attribute used to disambiguate local time when necessary. Many functions in the datetime have been updated to support local time disambiguation. See Local Time Disambiguation section for more information. (Contributed by Alexander Belopolsky in bpo-24773.)
The datetime.strftime() and date.strftime() methods now support ISO 8601 date directives %G, %u and %V. (Contributed by Ashley Anderson in bpo-12006.)
The datetime.isoformat() function now accepts an optional timespec argument that specifies the number of additional components of the time value to include. (Contributed by Alessandro Cucci and Alexander Belopolsky in bpo-19475.)
The datetime.combine() now accepts an optional tzinfo argument. (Contributed by Alexander Belopolsky in bpo-27661.) decimal
New Decimal.as_integer_ratio() method that returns a pair (n, d) of integers that represent the given Decimal instance as a fraction, in lowest terms and with a positive denominator:
Decimal('-3.14').as_integer_ratio()(-157, 50)
(Contributed by Stefan Krah amd Mark Dickinson in bpo-25928.)

## distutils

The default_format attribute has been removed from distutils.command.sdist.sdist and the formats attribute defaults to ['gztar']. Although not anticipated, any code relying on the presence of default_format may need to be adapted. See bpo-27819 for more details.
The upload command now longer tries to change CR end-of-line characters to CRLF. This fixes a corruption issue with sdists that ended with a byte equivalent to CR. (Contributed by Bo Bayles in bpo-32304.)

## email

The new email API, enabled via the policy keyword to various constructors, is no longer provisional. The email documentation has been reorganized and rewritten to focus on the new API, while retaining the old documentation for the legacy API. (Contributed by R. David Murray in bpo-24277.) The email.mime classes now all accept an optional policy keyword. (Contributed by Berker Peksag in bpo-27331.)
The DecodedGenerator now supports the policy keyword.
There is a new policy attribute, message_factory, that controls what class is used by default when the parser creates new message objects. For the email.policy.compat32 policy this is Message, for the new policies it is EmailMessage. (Contributed by R. David Murray in bpo-20476.)

## encodings

On Windows, added the 'oem' encoding to use CP_OEMCP, and the 'ansi' alias for the existing 'mbcs' encoding, which uses the CP_ACP code page. (Contributed by Steve Dower in bpo-27959.)

## enum

Two new enumeration base classes have been added to the enum module: Flag and IntFlags. Both are used to define constants that can be combined using the bitwise operators. (Contributed by Ethan Furman in bpo-23591.)
Many standard library modules have been updated to use the IntFlags class for their constants. The new enum.auto value can be used to assign values to enum members automatically:
from enum import Enum, auto>>> class Color(Enum):... red = auto()...
blue = auto()... green = auto()...>>> list(Color)[, , ] faulthandler
On Windows, the faulthandler module now installs a handler for Windows exceptions: see faulthandler.enable(). (Contributed by Victor Stinner in bpo-23848.)

## fileinput

hook_encoded() now supports the errors argument. (Contributed by Joseph Hackman in bpo-25788.)

## hashlib

hashlib supports OpenSSL 1.1.0. The minimum recommend version is 1.0.2. (Contributed by Christian Heimes in bpo-26470.)
BLAKE2 hash functions were added to the module. blake2b() and blake2s() are always available and support the full feature set of BLAKE2. (Contributed by Christian Heimes in bpo-26798 based on code by Dmitry Chestnykh and Samuel Neves. Documentation written by Dmitry Chestnykh.)
The SHA-3 hash functions sha3_224(), sha3_256(), sha3_384(), sha3_512(), and SHAKE hash functions shake_128() and shake_256() were added. (Contributed by Christian Heimes in bpo-16113. Keccak Code Package by Guido Bertoni, Joan Daemen, Michaël Peeters, Gilles Van Assche, and Ronny Van Keer.)
The password-based key derivation function scrypt() is now available with OpenSSL 1.1.0 and newer. (Contributed by Christian Heimes in bpo-27928.)

## http.client

HTTPConnection.request() and endheaders() both now support chunked encoding request bodies. (Contributed by Demian Brecht and Rolf Krahl in bpo-12319.)

## idlelib and IDLE

The idlelib package is being modernized and refactored to make IDLE look and work better and to make the code easier to understand, test, and improve. Part of making IDLE look better, especially on Linux and Mac, is using ttk widgets, mostly in the dialogs. As a result, IDLE no longer runs with tcl/tk 8.4. It now requires tcl/tk 8.5 or 8.6. We recommend running the latest release of either.
‘Modernizing’ includes renaming and consolidation of idlelib modules. The renaming of files with partial uppercase names is similar to the renaming of, for instance, Tkinter and TkFont to tkinter and tkinter.font in 3.0. As a result, imports of idlelib files that worked in 3.5 will usually not work in 3.6. At least a module name change will be needed (see idlelib/README.txt), sometimes more. (Name changes contributed by Al Swiegart and Terry Reedy in bpo-24225. Most idlelib patches since have been and will be part of the process.)
In compensation, the eventual result with be that some idlelib classes will be easier to use, with better APIs and docstrings explaining them. Additional useful information will be added to idlelib when available.

## importlib

Import now raises the new exception ModuleNotFoundError (subclass of ImportError) when it cannot find a module. Code that current checks for ImportError (in try-except) will still work. (Contributed by Eric Snow in bpo-15767.)
importlib.util.LazyLoader now calls create_module() on the wrapped loader, removing the restriction that importlib.machinery.BuiltinImporter and importlib.machinery.ExtensionFileLoader couldn’t be used with importlib.util.LazyLoader. importlib.util.cache_from_source(), importlib.util.source_from_cache(), and importlib.util.spec_from_file_location() now accept a path-like object.

## inspect

The inspect.signature() function now reports the implicit .0 parameters generated by the compiler for comprehension and generator expression scopes as if they were positional-only parameters called implicit0. (Contributed by Jelle Zijlstra in bpo-19611.)
To reduce code churn when upgrading from Python 2.7 and the legacy inspect.getargspec() API, the previously documented deprecation of inspect.getfullargspec() has been reversed. While this function is convenient for single/source Python 2/3 code bases, the richer inspect.signature() interface remains the recommended approach for new code. (Contributed by Nick Coghlan in bpo-27172)

## json

json.load() and json.loads() now support binary input. Encoded JSON should be represented using either UTF-8, UTF-16, or UTF-32. (Contributed by Serhiy Storchaka in bpo-17909.) logging
The new WatchedFileHandler.reopenIfNeeded() method has been added to add the ability to check if the log file needs to be reopened. (Contributed by Marian Horban in bpo-24884.)

## math

The tau (τ) constant has been added to the math and cmath modules. (Contributed by Lisa Roach in bpo-12345, see PEP 628 for details.)

## multiprocessing

Proxy Objects returned by multiprocessing.Manager() can now be nested. (Contributed by Davin Potts in bpo-6766.)

## os

See the summary of PEP 519 for details on how the os and os.path modules now support path-like objects.
scandir() now supports bytes paths on Windows.
A new close() method allows explicitly closing a scandir() iterator. The scandir() iterator now supports the context manager protocol. If a scandir() iterator is neither exhausted nor explicitly closed a ResourceWarning will be emitted in its destructor. (Contributed by Serhiy Storchaka in bpo-25994.) On Linux, os.urandom() now blocks until the system urandom entropy pool is initialized to increase the security. See the PEP 524 for the rationale.
The Linux getrandom() syscall (get random bytes) is now exposed as the new os.getrandom() function. (Contributed by Victor Stinner, part of the PEP 524)

## pathlib

pathlib now supports path-like objects. (Contributed by Brett Cannon in bpo-27186.) See the summary of PEP 519 for details.

## pdb

The Pdb class constructor has a new optional readrc argument to control whether .pdbrc files should be read.

## pickle

Objects that need new called with keyword arguments can now be pickled using pickle protocols older than protocol version 4. Protocol version 4 already supports this case. (Contributed by Serhiy Storchaka in bpo-24164.)

## pickletools

pickletools.dis() now outputs the implicit memo index for the MEMOIZE opcode. (Contributed by Serhiy Storchaka in bpo-25382.)

## pydoc

The pydoc module has learned to respect the MANPAGER environment variable. (Contributed by Matthias Klose in bpo-8637.)
help() and pydoc can now list named tuple fields in the order they were defined rather than alphabetically. (Contributed by Raymond Hettinger in bpo-24879.)

## random

The new choices() function returns a list of elements of specified size from the given population with optional weights. (Contributed by Raymond Hettinger in bpo-18844.)

## re

Added support of modifier spans in regular expressions. Examples: '(?i:p)ython' matches 'python' and 'Python', but not 'PYTHON'; '(?i)g(?-i:v)r' matches 'GvR' and 'gvr', but not 'GVR'. (Contributed by Serhiy Storchaka in bpo-433028.)
Match object groups can be accessed by getitem, which is equivalent to group(). So mo['name'] is now equivalent to mo.group('name'). (Contributed by Eric Smith in bpo-24454.)
Match objects now support index-like objects as group indices. (Contributed by Jeroen Demeyer and
Xiang Zhang in bpo-27177.)

Added set_auto_history() to enable or disable automatic addition of input to the history list. (Contributed by Tyler Crompton in bpo-26870.)

## rlcompleter

Private and special attribute names now are omitted unless the prefix starts with underscores. A space or a colon is added after some completed keywords. (Contributed by Serhiy Storchaka in bpo-25011 and bpo-25209.)

## shlex

The shlex has much improved shell compatibility through the new punctuation_chars argument to control which characters are treated as punctuation. (Contributed by Vinay Sajip in bpo-1521950.) site
When specifying paths to add to sys.path in a .pth file, you may now specify file paths on top of directories (e.g. zip files). (Contributed by Wolfgang Langner in bpo-26587). sqlite3
sqlite3.Cursor.lastrowid now supports the REPLACE statement. (Contributed by Alex LordThorsen in bpo-16864.)

## socket

The ioctl() function now supports the SIO_LOOPBACK_FAST_PATH control code. (Contributed by Daniel Stokes in bpo-26536.)
The getsockopt() constants SO_DOMAIN, SO_PROTOCOL, SO_PEERSEC, and SO_PASSSEC are now supported. (Contributed by Christian Heimes in bpo-26907.)
The setsockopt() now supports the setsockopt(level, optname, None, optlen: int) form. (Contributed by Christian Heimes in bpo-27744.)
The socket module now supports the address family AFALG to interface with Linux Kernel crypto API. ALG*, SOL_ALG and sendmsg_afalg() were added. (Contributed by Christian Heimes in bpo-27744 with support from Victor Stinner.)
New Linux constants TCP_USER_TIMEOUT and TCP_CONGESTION were added. (Contributed by Omar Sandoval, issue:26273).

## socketserver

Servers based on the socketserver module, including those defined in http.server, xmlrpc.server and wsgiref.simple_server, now support the context manager protocol. (Contributed by Aviv Palivoda in bpo-26404.)
The wfile attribute of StreamRequestHandler classes now implements the io.BufferedIOBase writable interface. In particular, calling write() is now guaranteed to send the data in full. (Contributed by Martin Panter in bpo-26721.)

## ssl

ssl supports OpenSSL 1.1.0. The minimum recommend version is 1.0.2. (Contributed by Christian Heimes in bpo-26470.)
3DES has been removed from the default cipher suites and ChaCha20 Poly1305 cipher suites have been added. (Contributed by Christian Heimes in bpo-27850 and bpo-27766.)
SSLContext has better default configuration for options and ciphers. (Contributed by Christian Heimes in bpo-28043.)
SSL session can be copied from one client-side connection to another with the new SSLSession class. TLS session resumption can speed up the initial handshake, reduce latency and improve performance (Contributed by Christian Heimes in bpo-19500 based on a draft by Alex Warhawk.)
The new get_ciphers() method can be used to get a list of enabled ciphers in order of cipher priority. All constants and flags have been converted to IntEnum and IntFlags. (Contributed by Christian Heimes in bpo-28025.)
Server and client-side specific TLS protocols for SSLContext were added. (Contributed by Christian Heimes in bpo-28085.)

## statistics

A new harmonic_mean() function has been added. (Contributed by Steven D’Aprano in bpo-27181.)

## struct

struct now supports IEEE 754 half-precision floats via the 'e' format specifier. (Contributed by Eli Stevens, Mark Dickinson in bpo-11734.)

## subprocess

subprocess.Popen destructor now emits a ResourceWarning warning if the child process is still running. Use the context manager protocol (with proc: ...) or explicitly call the wait() method to read the exit status of the child process. (Contributed by Victor Stinner in bpo-26741.) The subprocess.Popen constructor and all functions that pass arguments through to it now accept encoding and errors arguments. Specifying either of these will enable text mode for the stdin, stdout and stderr streams. (Contributed by Steve Dower in bpo-6135.)

## sys

The new getfilesystemencodeerrors() function returns the name of the error mode used to convert between Unicode filenames and bytes filenames. (Contributed by Steve Dower in bpo-27781.) On Windows the return value of the getwindowsversion() function now includes the platform_version field which contains the accurate major version, minor version and build number of the current operating system, rather than the version that is being emulated for the process (Contributed by Steve Dower in bpo-27932.)

##### Image signatures are necessary, but not adequate. Here's why you can't rely on them alone

From: Signed, Sealed, Deployed: Why Signatures Aren't Enough
Red Hat recently blogged about their progress in adding support for container image signing, a particularly interesting and most welcome aspect of the design is the way that the binary signature file can be decoupled from the registry and distributed separately. The blog makes interesting reading and I’d strongly recommending reading through it, I’m sure you’ll appreciate the design. And of course code is available on line.
Red Hat is, along with the other Linux distributors, well versed in the practice of signing software components to allow end users to verify that they are running authentic code and is in the process of extending this support to container images. The approach described is different to that taken previously by Docker Inc. however rather than comparing the two approaches I wanted to talk at a high level about the benefits of image signing along with some commentary about trust.
In the physical world we are all used to using our signature to confirm our identity. Probably the most common example is when we are signing a paper check or using an electronic signature pad during a sales transaction. How many times have you signed your signature so quickly that you do not even recognize it yourself? How many times in recent memory has a cashier or server compared the signature written with that on the back of your credit card? In my experience that check is likely to happen one out of every ten times and even in those cases that check is little more than a token gesture and the two signatures may not have matched.
That leads me to the first important observation: a signature mechanism is only useful if it is checked. Obviously when vendors such as Docker Inc, Red Hat and others implement an image signing and validation system, the enforcement will be built into to all layers, so that in one example a Red Hat delivered image will be validated by a Red Hat provided Docker runtime to ensure it’s signed by a valid source.
However it’s more likely that the images that you deploy in your enterprise won’t just be the images downloaded from a registry, but instead images built on top of these images or perhaps even built from scratch, so for image signing to provide the required level of security all images created within your enterprise should also be signed and have those signatures validated before the image is deployed. Some early users of image signing that we have talked to have used image signing less as a way of tracking provenance of images but instead as a method to show that an image has not been modified between leaving the CI/CD pipeline and being deployed on their container host.
Before we dig into the topic of image signing it’s worth discussing what a signature actually represents. The most common example of signatures that we see in our day to day life is in our web browsers where we look for the little green padlock in address bar that indicates that the connection to the web server from our browser is encrypted but most importantly it confirms that you are talking to the expected website.
The use of TLS/SSL certificates allows your browser to validate that when you connect to https://www.example.com the content displayed actually came from example.com. So in this example the signature was used to confirm the source of the (web) content. Over many years we have been trained NOT to type our credit card details into a site that is NOT delivered through HTTPS. But that does not mean that you would trust your credit card details to any site that uses HTTPS.
The same principle applies to the use of image signatures. If you download an image signed by Red Hat, Docker Inc, or any other vendor, you can be assured that the image did come from this vendor. The level of confidence you have in the contents of the image is based on the level of trust you already have with the vendor. For example you are likely not to run an image signed by l33thackerz even though it may include a valid signature. As enterprises move to a DevOps model with containers we’re seeing a new software supply chain, which often begins with a base image pulled from DockerHub or a vendor registry. This base image may be modified by the operations team to include extra packages or to customize specific configuration files. The resulting image is then published in the local registry to be used by the development team as the base image for their application container. In many organizations we are starting to see other participants in this supply chain, for example a middleware team may publish an image containing an application server that is in turn used by an application team.
For the promise of image signing to be fulfilled, at each stage of this supply chain each team must sign the image to ensure that the ‘chain of custody’ can be validated throughout the software development lifecycle. As we covered previously those signatures only serves to prove the source of an image, during any point in the supply chain from the original vendor of the base image all the way through the development process the images may be modified. At any step in the supply chain a mistake may be made, an outdated package that contains known bugs of vulnerabilities may be used, an insecure configuration option in an application’s configuration file, or perhaps secrets such as passwords or API keys may be stored in the image.
Signing an image will not prevent insecure or otherwise non-compliant images from being deployed, however as part of a post mortem it will provide a way of tracking down when the vulnerability or bug was introduced.
During each stage of the supply chain detailed checks should be performed on the image to ensure that the image complies with your site specific policies.
These policies could cover security, starting with the ubiquitous CVE scan but then going further to analyze configuration of key security components. For example, you could have the latest version of the Apache web server but have configured the wrong set of TLS Ciphers suites leading to insecure communication. In addition to security, your policies could cover application specific configurations to comply with best practices or to enable consistency and predictability.
Anchore’s goal is to provide a toolset that allows developers, operations, and security teams to maintain full visibility of the ‘chain of custody’ as containers move through the development lifecycle, while providing the visibility, predictability, and control needed for production deployment.
With Anchore’s tools the analysis and policy evaluation could be run during each stage of the supply chain allowing the signatures to attest to both the source of the image and also the compliance of the image’s contents.
In summary, we believe that image signing is an important part of the security and integrity of your software supply chain however signatures alone will not ensure the integrity of your systems.