The solution in this answer has been incorporated into ruamel.yaml using a plugin mechanism. At the bottom of this post there are quick-and-dirty instructions on how to use that.
There are three aspects in updating a YAML file that contains jinja2 “code”:
- making the jinja2 code acceptable to the YAML parser
- making sure the acceptable can reversed (i.e. the changes should be unique, so only they get reversed)
- preserving the layout of the YAML file so that the updated file processed by jinja2 still produces a valid YAML file, that again can be loaded.
Let’s start by making your example somewhat more realistic by adding a jinja2 variable definition and for-loop and adding some comments (input.yaml):
# trying to update
{% set xyz = "123" }
A:
B:
- ip: 1.2.3.4
- myArray:
- {{ jinja.variable }}
- val1
- val2 # add a value after this one
{% for d in data %}
- phone: {{ d.phone }}
name: {{ d.name }}
{% endfor %}
- {{ xyz }}
# #% or ##% should not be in the file and neither <{ or <<{
The lines starting with {% contain no YAML, so we’ll make those into comments (assuming that comments are preserved on round-trip, see below). Since YAML scalars cannot start with { without being quoted we’ll change the {{ to <{. This is done in the following code by calling sanitize() (which also stores the patterns used, and the reverse is done in sanitize.reverse (using the stored patterns).
The preservation of your YAML code (block-style etc) is best done using ruamel.yaml (disclaimer: I am the author of that package), that way you don’t have to worry about flow-style elements in the input getting mangled into as block style as with the rather crude default_flow_style=False that the other answers use. ruamel.yaml also preserves comments, both the ones that were originally in the file, as well as those temporarily inserted to “comment out” jinja2 constructs starting with %{.
The resulting code:
import sys
from ruamel.yaml import YAML
yaml = YAML()
class Sanitize:
"""analyse, change and revert YAML/jinja2 mixture to/from valid YAML"""
def __init__(self):
self.accacc = None
self.accper = None
def __call__(self, s):
len = 1
for len in range(1, 10):
pat="<" * len + '{'
if pat not in s:
self.accacc = pat
break
else:
raise NotImplementedError('could not find substitute pattern '+pat)
len = 1
for len in range(1, 10):
pat="#" * len + '%'
if pat not in s:
self.accper = pat
break
else:
raise NotImplementedError('could not find substitute pattern '+pat)
return s.replace('{{', self.accacc).replace('{%', self.accper)
def revert(self, s):
return s.replace(self.accacc, '{{').replace(self.accper, '{%')
def update_one(file_name, out_file_name=None):
sanitize = Sanitize()
with open(file_name) as fp:
data = yaml.load(sanitize(fp.read()))
myArray = data['A']['B'][1]['myArray']
pos = myArray.index('val2')
myArray.insert(pos+1, 'val 3')
if out_file_name is None:
yaml.dump(data, sys.stdout, transform=sanitize.revert)
else:
with open(out_file_name, 'w') as fp:
yaml.dump(data, out, transform=sanitize.revert)
update_one('input.yaml')
which prints (specify a second parameter to update_one() to write to a file) using Python 2.7:
# trying to update
{% set xyz = "123" }
A:
B:
- ip: 1.2.3.4
- myArray:
- {{ jinja.variable }}
- val1
- val2 # add a value after this one
- val 3
{% for d in data %}
- phone: {{ d.phone }}
name: {{ d.name }}
{% endfor %}
- {{ xyz }}
# #% or ##% should not be in the file and neither <{ or <<{
If neither #{ nor <{ are in any of the original inputs then sanitizing and reverting can be done with simple one-line functions (see this versions of this post), and then you don’t need the class Sanitize
Your example is indented with one position (key B) as well as two positions (the sequence elements), ruamel.yaml doesn’t have that fine control over output indentation (and I don’t know of any YAML parser that does). The indent (defaulting to 2) is applied to both YAML mappings as to sequence elements (measured to the beginning of the element, not to the dash). This has no influence on re-reading the YAML and happened to the output of the other two answerers as well (without them pointing out this change).
Also note that YAML().load() is safe (i.e. doesn’t load arbitrary potentially malicious objects), whereas the yaml.load() as used in the other answers is definitely unsafe, it says so in the documentation and is even mentioned in the WikiPedia article on YAML. If you use yaml.load(), you would have to check each and every input file to make sure there are no tagged objects that could cause your disc to be wiped (or worse).
If you need to update your files repeatedly, and have control over the jinja2 templating, it might be better to change the patterns for jinja2 once and not revert them, and then specifying appropriate block_start_string, variable_start_string (and possible block_end_string and variable_end_string) to the jinja2.FileSystemLoader added as loader to the jinja2.Environment.
If the above seems to complicated then in a a virtualenv do:
pip install ruamel.yaml ruamel.yaml.jinja2
assuming you have the input.yaml from before you can run:
import os
from ruamel.yaml import YAML
yaml = YAML(typ='jinja2')
with open('input.yaml') as fp:
data = yaml.load(fp)
myArray = data['A']['B'][1]['myArray']
pos = myArray.index('val2')
myArray.insert(pos+1, 'val 3')
with open('output.yaml', 'w') as fp:
yaml.dump(data, fp)
os.system('diff -u input.yaml output.yaml')
to get the diff output:
--- input.yaml 2017-06-14 23:10:46.144710495 +0200
+++ output.yaml 2017-06-14 23:11:21.627742055 +0200
@@ -8,6 +8,7 @@
- {{ jinja.variable }}
- val1
- val2 # add a value after this one
+ - val 3
{% for d in data %}
- phone: {{ d.phone }}
name: {{ d.name }}
ruamel.yaml 0.15.7 implements a new plug-in mechanism and ruamel.yaml.jinja2 is a plug-in that rewraps the code in this answer transparently for the user. Currently the information for reversion is attached to the YAML() instance, so make sure you do yaml = YAML(typ='jinja2') for each file you process (that information could be attached to the top-level data instance, just like the YAML comments are).