Separate Data from HTML and put into js _data file

Ed H AHSAA Blog
Ed's Blog Index & Contents

Sep 7, 2022 7:00

Separate data from HTML for approximately 100 data sets (newsletters) and move the data into a JS (javascript) _data file

  1. AmesHigh.org had a severly long newsletter.html file that grew over 20 years and filled with lots and lots of info, filenames, file sizes, volume numbers etc. from almost 100 published Newsletters. The data was jumbled and mixed together into one HTML file. Data intermixed with HTML. It was hard to read, hard to keep the resulting web page looking nice because every one of the 94 newsletters had its own HTML. The good news was it was in an HTML Table, so every piece of data was between a ending and beginning tag.

  2. I Separated and removed all the data from this file and put it into a js _data file which I named allNewsletters.js

  3. Took awhile !! Used the atom editor find and replace_all function many times to speed up the process of decoupling the data from the html. The good news was each piece of data was inside and tags which made it slightly easy to do a search all and replace all. I replaced some of the with something like "size": " and then the ending with something like ",

  4. Originally, newsletter.html was built with a large table with 94 rows of newsletter info and data. It would soon be over 100 different table entries so I decided to pull the plug and separate the data before it got any larger.

    The original newsletter.html was in the following format. For example this is the data and html from the 76th newsletter published August 2017. There were 93 other newsletters that looked about like this with all different data.

<tr>
	<th>76</th>
	<td><a href="newsletter/2017/2017_Aug_Vol_28_issue2_AHSAA_newsletter_2017_Scholarship_winners_164964.pdf" target="_blank">
			2017</a></td>
	<td>Aug</td>
	<td>v 28</td>
	<td>i 2</td>
	<td>3.4 MB</td>
	<td>PDF</td>
	<td>
		<ul class="noindent">
			<li><strong>Aug 2017 AHSAA newsletter</strong></li>
			<li>AHS Class of 1992 25th Reunion</li>
			<li>9 class reunions this summer, all 2s and 7s.
				1952, 1957, 1962, 1967, 1972, 1977, 1982, 1992 and 2007.</li>
		</ul>
	</td>
</tr>

This was cumbersome HTML. Everytime I needed to add or update an AHSAA newsletter I had to figure out the format again. 3 new newsletters per year.

Starting the process to change the HTML file to a JS file

I was able to global search for patterns of tags like <td>i 2</td> and global replace with "issue": "2", or globally searched for <td>Aug</td> and replaced with "month": "August",then searched for <td>Sep</td> and replaced with "month": "September", etc. I Did this for all 12 months. So I had 12 different searches and replacements just for the months.

Still, even with being able to automate a little of the change, it took awhile and had to manually change a lot of the file.

Version 1 of new newsletter data file format looks like this.

First iteration. The description area below was not working out as easily or nicely as I thought in this first attempt at separating the data from the HTML.
{
"filename" : "2017_Aug_Vol_28_issue2_AHSAA_newsletter_2017_Scholarship_winners_164964.pdf",
"year": "2017",
"month": "August",
"volume": "28",
"issue": "2",
"size": "3.4",
"type": "PDF",
"description": " \
<ol class='pl-5 is-small content is-type-1'> \
  <li><strong>Aug 2017 AHSAA newsletter</strong></li> \
  <li>AHS Class of 1992 25th Reunion</li> \
  <li>9 class reunions this summer, all 2s and 7s. \
    1952, 1957, 1962, 1967, 1972, 1977, 1982, 1992 and 2007.</li> \
</ol> \
",
},

For the description part of the data, I tried both an ordered list and unordered list and neither worked well, plus it was still tedius to put each line inside of a

  • list tag and I still had HTML mixed in with my data which ulitmatly I did not want any HTML left inside my data. I bit the bullet again and changed the description area to js also and on to version 2.

    Version 2 of the new newsletter data file format. I changed the description to also be an array and removed all the HTML from the data as you can see
    {
"filename" : "2012_Dec_Vol_23_Issue_3_AHSAA_newsletter_143982.pdf",
"year": "2012",
"month": "August",
"volume": "23",
"issue": "3",
"size": "2.8",
"type": "PDF",
"description": [
// description = changed to an array Sep 9 2022 Ed H
   [ "Architect Richard B. (Dick) Campbell 1955 named Distinguished Alum 2012 "],
   [ "2012 Ames High School Athletic Hall of Fame Inductees: : John Sletten Coach, Bonnie Gagnier Beckerich 1979, \
    Lee Sargent AHS 1964, Steve Dreyer AHS 1987, Doran Geise AHS 1978, Lloyd Kester AHS 1942 "],
   [ "Growing up in Ames - Dennis Tice, AHS 1960 "],
   [ "Some Have Left US: 1967 Lowell Johannes d.10-16-2012 Denver,CO, 1969 Joellyn (Borke) Johnston d.6-11-2011 Des Moines,IA, \
    1969 Mark Randolph Ketcham d.10-8-2012 Arlington TX, 1975 Kerry Ferguson d.10-2-2012 Rochester MN, \
    1975 Lisa (Thurston) Hotchkiss d.9-27-2012 Austin,TX "],
   [ "AHS Class of 1972 40th Reunion "],
   [ "Other News About Graduates "],
   [ "AHS Class of 1954 Tweener! "],
   [ "Two-Wheeling Through the Canadian Rockies - Bill Burke 1973, Gary Clem 1961, Steve Risdal 1965, Linda Speck 1969, \
    Mark Speck 1969 and Jim Olson 1973 "],
   [ "AHS 1955 Celebrate 75th Bday "],
  ]
   // description is now an array as of Sep 8 2022 so it can be controlled from the newsletter.njk file
},

    I named this data file allNewsletters.js There are 94 entries or arrays in the allNewsletters.js data file which is located in the Eleventy _data folder. It took a while, longer than I expected to split the original HTML into a data file and a template file. To my defense, I also tweaked the data and the template code and was experimenting with different template layouts and different Bulma styles, so that took some time too. The data is now completely separated from the HTML so I can change easily tweak template code which changes how all 94 entries are displayed with one click. In the old days, I would have to edit all 94 HTML entries and that was not going to happen. The page was going to look worse and worse. Well, not anymore!

    Now that the _data is in a JS style file it is super fun and super easy to improve the look and feel of the page, and super easy for Eleventy to access this data.

    How to access a javascript js data file from Eleventy and Nunjucks

    To access the data file that I just created above, I used the following Nunjucks for loop. This iterates thru all the data. The code below is inside of a Nunjucks file that I named newsletter.njk

    {%- for newsletter in allNewsletters -%}
      {# for loop to iterate thru all 33 years of newsletters #}

    Every newsletter has its own set of infomation, for instance, year {{newsletter.year}}, issue # {{newsletter.issue}}, size {{newsletter.size}}, etc. Inside the for loop I can display the data {{newsletter.something}} however I like using HTML and CSS (Bulma.io). For example, further on down in the newletter.njk file inside the above for loop, you see this code.

    <span id="news{{newsletter.year}}"></span>
    {# ID to navigate to of the form #newsYYYY for each year drom the dropdown include below #}

<div class="message is-success mt-5">
  {# Each newsletter volume (an entire year of newsletters) gets its own Bulma message #}
  <div class=" message-header is-size-3 is-size-6-mobile">
    {{ newsletter.year }}
    <span class="is-size-4 is-size-7-mobile">Volume
      {{newsletter.volume}}</span>
    {% include "_all_newsletter_years_dropdown.html" %}
    {# include -- displays a long dropdown list of all the newsletter years #}
  </div>
</div>

    How to access the data array, which is, in turn, inside another array in the js _data file

    The array description is nested inside another array in the datafile allNewsletters.js and is accessable with the following for loop which is in turn nested inside the above for loop.

    {%- for desc in newsletter.description -%}

    Inside the above for loop you access each line of the description array as the desc variable for each time thru the loop. For example:

    <a href="{% if newsletter.filename != 'membership.html' %}{{newsletter.year}}{% endif %}/{{newsletter.filename}}" target="_blank">
  <label class="panel-block is-size-6 is-size-7-mobile">
    <span class="panel-icon">
      <i class="fa-regular fa-newspaper"></i>
    </span>
    <span class='has-text-weight-medium'> {# there is a bug I found in Bulma that has trouble rendering the ahs or span inside of a span or maybe insite of a lable class anyway it is fixed if I put it inside of a has-text-weight-bold or -medium class figured this out 9/10/2022 been fighting it for awhile Ed Hendrickson  #}
    {{desc | safe
  | replace("AHSAA" , "<span class='orange'>AHS</span>AA")
  | replace(" AHS "   , " <span class='orange'>AHS</span> ")
}}
</span>

  </label>
</a>

    Side Note

    The Nunjucks replace function is super nice and useful. I was tired of typing <span class="orange">AHS</span> over and over hundreds of times. Now I type AHS and the replace function you see in the above code will automatically change it for me to the HTML code I want.

    Nunjucks replace function documentation

    Want to see the result of separating data from HTML and using Eleventy powered templating. Click here

    •