Jekyll2023-08-23T04:25:53+00:00https://rusyasoft.github.io/atom.xmlWelcome to Rustam’s BlogRustam Rakhimov IgorevichTesting Complex Logic Within Task Runners in Java2023-08-22T00:00:00+00:002023-08-22T00:00:00+00:00https://rusyasoft.github.io/java/2023/08/22/Unit-test-logics-under-taskrunner<p>In this blog post, we’ll explore a common challenge faced by Java developers - testing complex logic within task runners. We’ll discuss a practical approach to isolate and test the logic within task runners effectively.</p>
<h1 id="the-challenge">The Challenge</h1>
<p>Consider a scenario where you have a codebase structured with a TaskRunner-like pattern, encapsulating various tasks. For instance:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">Measurer</span> <span class="o">{</span>
<span class="kd">public</span> <span class="o"><</span><span class="no">T</span><span class="o">></span> <span class="no">T</span> <span class="nf">performanceMeasurerTaskRunner</span><span class="o">(</span><span class="nc">Supplier</span><span class="o"><</span><span class="no">T</span><span class="o">></span> <span class="n">supplier</span><span class="o">)</span> <span class="o">{</span>
<span class="kt">long</span> <span class="n">startTime</span> <span class="o">=</span> <span class="nc">System</span><span class="o">.</span><span class="na">currentTimeMillis</span><span class="o">();</span>
<span class="no">T</span> <span class="n">res</span> <span class="o">=</span> <span class="n">supplier</span><span class="o">.</span><span class="na">get</span><span class="o">();</span>
<span class="kt">long</span> <span class="n">endTime</span> <span class="o">=</span> <span class="nc">System</span><span class="o">.</span><span class="na">currentTimeMillis</span><span class="o">();</span>
<span class="kt">long</span> <span class="n">runningTime</span> <span class="o">=</span> <span class="n">endTime</span> <span class="o">-</span> <span class="n">startTime</span><span class="o">;</span>
<span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"Task performed in "</span> <span class="o">+</span> <span class="n">runningTime</span> <span class="o">+</span> <span class="s">" milliseconds"</span><span class="o">);</span>
<span class="k">return</span> <span class="n">res</span><span class="o">;</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Now, let’s say you want to test the logic within the usefulTask function without being encumbered by the performance measurement code. We’ll explore a solution that allows us to isolate and test usefulTask independently.</p>
<h1 id="the-solution">The Solution</h1>
<p>To address this issue, we can mock the Measurer class, particularly the performanceMeasurerTaskRunner method, so that it bypasses the performance measurement logic. Here’s how you can do it:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">@Mock</span>
<span class="kd">private</span> <span class="nc">Measurer</span> <span class="n">mockedMeasurer</span><span class="o">;</span>
<span class="nd">@InjectMocks</span>
<span class="kd">private</span> <span class="nc">MainClass</span> <span class="n">mainClass</span><span class="o">;</span>
<span class="kd">private</span> <span class="kt">void</span> <span class="nf">mockTaskRunnerWithPassThrough</span><span class="o">()</span> <span class="o">{</span>
<span class="n">when</span><span class="o">(</span><span class="n">mockedMeasurer</span><span class="o">.</span><span class="na">performanceMeasurerTaskRunner</span><span class="o">(</span><span class="n">any</span><span class="o">(</span><span class="nc">Supplier</span><span class="o">.</span><span class="na">class</span><span class="o">))).</span><span class="na">thenAnswer</span><span class="o">(</span><span class="n">invocation</span> <span class="o">-></span> <span class="o">{</span>
<span class="nc">Supplier</span><span class="o"><?></span> <span class="n">supplier</span> <span class="o">=</span> <span class="n">invocation</span><span class="o">.</span><span class="na">getArgument</span><span class="o">(</span><span class="mi">0</span><span class="o">);</span>
<span class="k">return</span> <span class="n">supplier</span><span class="o">.</span><span class="na">get</span><span class="o">();</span>
<span class="o">});</span>
<span class="o">}</span>
<span class="nd">@Test</span>
<span class="kt">void</span> <span class="nf">testUsefulTask</span><span class="o">()</span> <span class="o">{</span>
<span class="n">mockTaskRunnerWithPassThrough</span><span class="o">();</span>
<span class="n">mainClass</span><span class="o">.</span><span class="na">usefulJob</span><span class="o">();</span>
<span class="o">}</span>
</code></pre></div></div>
<p>In the code above, we configure the mockedMeasurer to pass through any calls to its underlying Supplier.</p>
<h1 id="benefits-of-the-approach">Benefits of the Approach</h1>
<p><strong>Isolation</strong>: This approach allows us to isolate and test specific logic within the task runner, ignoring unrelated complexities.</p>
<p><strong>Simplicity</strong>: It simplifies the testing process, especially when dealing with complex dependencies such as databases or external APIs.</p>
<h1 id="conclusion">Conclusion</h1>
<p>Testing complex logic within task runners in Java can be challenging, but by employing this technique, you can focus your testing efforts on the specific logic you need to validate. This method provides flexibility and maintainability, ensuring your tests remain concise and targeted, even in complex codebases.</p>
<p>In conclusion, while it’s ideal to separate logic into independent functions for testing, this approach offers a practical solution for scenarios where such separation is not feasible. By understanding and utilizing the power of mocking, you can streamline your testing process and maintain code quality effectively.</p>
<p>We encourage you to apply these techniques in your Java projects and share your experiences in the comments section below. If you have any questions or need further clarification, feel free to ask.</p>Rustam Rakhimov IgorevichIn this blog post, we’ll explore a common challenge faced by Java developers - testing complex logic within task runners. We’ll discuss a practical approach to isolate and test the logic within task runners effectively.Rollout Logics Wrapper2023-06-25T00:00:00+00:002023-06-25T00:00:00+00:00https://rusyasoft.github.io/java/2023/06/25/Rollout-Logics-Wrapper<p>In modern software development, the process extends beyond simply completing a task and moving on. It entails continuous work on new updates, features, and bug fixes. Therefore, the role of feature flags becomes crucial in the software’s lifecycle. Feature flags can be classified as either static or dynamic, depending on the speed at which their changes are applied to the application’s behavior. In this post, we will not discuss the various types of feature flags; instead, we will concentrate on the different approaches for writing code that depends on these variables and styles.</p>
<p>Let’s examine the following typical example, where we have a ProductService responsible for retrieving product details by productId from the productRepository. The RenderService, on the other hand, obtains product details from the ProductService and then proceeds to render the page.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">ProductService</span> <span class="o">{</span>
<span class="kd">private</span> <span class="nc">ProductRepository</span> <span class="n">productRepository</span><span class="o">;</span>
<span class="kd">public</span> <span class="nc">Product</span> <span class="nf">getProductDetails</span><span class="o">(</span><span class="nc">Long</span> <span class="n">productId</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="n">productRepository</span><span class="o">.</span><span class="na">find</span><span class="o">(</span><span class="n">productId</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="kd">class</span> <span class="nc">RenderService</span> <span class="o">{</span>
<span class="kd">private</span> <span class="nc">ProductService</span> <span class="n">producService</span><span class="o">;</span>
<span class="kd">public</span> <span class="nc">Page</span> <span class="nf">renderProduct</span><span class="o">(</span><span class="nc">Long</span> <span class="n">productId</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">Product</span> <span class="n">product</span> <span class="o">=</span> <span class="n">productService</span><span class="o">.</span><span class="na">getProductDetails</span><span class="o">(</span><span class="n">productId</span><span class="o">);</span>
<span class="c1">// Do a lot of other stuff required to render product page</span>
<span class="k">return</span> <span class="nf">renderFinalPage</span><span class="o">(</span><span class="n">product</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<h2 id="now-we-got-new-fast-product-repository">Now we got new Fast Product Repository</h2>
<p>The story continues: Suddenly, the platform team introduces a new type of database that is incredibly fast. Queries on this new database respond much quicker compared to our current solution. The platform team has conducted some experiments, confirming its speed, and now they suggest that we adapt to this new database. As the owner of our service, we become enamored with the idea and are eager to embrace it. However, we mustn’t forget that numerous other services also depend on us, and some of them rely on our libraries. We have a responsibility towards those who depend on us. If things were to go wrong later on, and it turns out that the Fast Repository isn’t as fast as anticipated or hasn’t been thoroughly tested in a real production network with significantly higher and different traffic patterns than those generated in the lab, we would bear the consequences.</p>
<p>So, we need to devise a rollout strategy for implementing the new database.</p>
<h2 id="quick-rollout-way">Quick Rollout Way</h2>
<p>Quick flag based rollout way would be the fastest and riskiest way of coming up. Example of it can be as follows and rely on application.properties value.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">ProductService</span> <span class="o">{</span>
<span class="kd">private</span> <span class="nc">ProductRepository</span> <span class="n">productRepository</span><span class="o">;</span>
<span class="kd">private</span> <span class="nc">FastProductRepository</span> <span class="n">fastProductRepository</span><span class="o">;</span>
<span class="nd">@Value</span><span class="o">(</span><span class="s">"${product.repo.fastFlag:false}"</span><span class="o">)</span>
<span class="kd">private</span> <span class="nc">Boolean</span> <span class="n">fastProductRepoFlag</span><span class="o">;</span>
<span class="kd">public</span> <span class="nc">Product</span> <span class="nf">getProductDetails</span><span class="o">(</span><span class="nc">Long</span> <span class="n">productId</span><span class="o">)</span> <span class="o">{</span>
<span class="k">if</span> <span class="o">(</span><span class="n">fastProductRepoFlag</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="n">fastProductRepository</span><span class="o">.</span><span class="na">find</span><span class="o">(</span><span class="n">productId</span><span class="o">);</span>
<span class="o">}</span>
<span class="k">return</span> <span class="n">productRepository</span><span class="o">.</span><span class="na">find</span><span class="o">(</span><span class="n">productId</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>The advantage of this approach is that the implementation time is shorter, and the deployment can be carried out almost immediately. However, the main disadvantage is the inherent risk involved. By relying on luck alone, once we enable the feature flag, we are taking a gamble. This assumes that we do not have platform-level DevOps practices in place, where instances can be gradually deployed by not enabling the flag for all instances at once. To mitigate this risk, it is crucial to have robust platform deployment controls and traffic management in place. These measures can help reduce the potential negative impact of any unforeseen issues.</p>
<h2 id="task-runner-based-rollout">Task Runner based Rollout</h2>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">enum</span> <span class="nc">ProductRolloutPhase</span> <span class="o">{</span>
<span class="no">FAST_PRODUCT_DISABLED</span><span class="o">,</span>
<span class="no">FAST_PRODUCT_DRY_RUN</span><span class="o">,</span>
<span class="no">FAST_PRODUCT_ENALBED</span><span class="o">;</span>
<span class="o">}</span>
<span class="kd">class</span> <span class="nc">RolloutTask</span> <span class="o">{</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="kd">final</span> <span class="nc">ProductRolloutPhase</span> <span class="no">DEFAULT_ROLLOUT_PHASE</span> <span class="o">=</span> <span class="nc">ProductRolloutPhase</span><span class="o">.</span><span class="na">FAST_PRODUCT_DISABLED</span><span class="o">;</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="nc">ThreadLocal</span><span class="o"><</span><span class="nc">Boolean</span><span class="o">></span> <span class="n">productRolloutThreadLocal</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">ThreadLocal</span><span class="o"><>();</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kt">boolean</span> <span class="nf">getCurrentFlag</span><span class="o">()</span> <span class="o">{</span>
<span class="k">if</span> <span class="o">(</span><span class="n">productRolloutThreadLocal</span><span class="o">.</span><span class="na">get</span><span class="o">()</span> <span class="o">==</span> <span class="kc">null</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="kc">false</span><span class="o">;</span>
<span class="o">}</span>
<span class="k">return</span> <span class="n">productRolloutThreadLocal</span><span class="o">.</span><span class="na">get</span><span class="o">();</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="o"><</span><span class="no">T</span><span class="o">></span> <span class="no">T</span> <span class="nf">execute</span><span class="o">(</span><span class="nc">ProductRolloutPhase</span> <span class="n">rolloutPhase</span><span class="o">,</span> <span class="nc">Supplier</span><span class="o"><</span><span class="no">T</span><span class="o">></span> <span class="n">delegator</span><span class="o">)</span> <span class="o">{</span>
<span class="kt">boolean</span> <span class="n">prevFlagValue</span> <span class="o">=</span> <span class="n">getCurrentFlag</span><span class="o">();</span>
<span class="k">try</span> <span class="o">{</span>
<span class="k">if</span> <span class="o">(</span><span class="n">rolloutPhase</span> <span class="o">==</span> <span class="no">FAST_PRODUCT_DISABLED</span><span class="o">)</span> <span class="o">{</span>
<span class="n">productRolloutThreadLocal</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="kc">false</span><span class="o">);</span>
<span class="o">}</span> <span class="k">else</span> <span class="k">if</span> <span class="o">(</span><span class="n">rolloutPhase</span> <span class="o">==</span> <span class="no">FAST_PRODUCT_ENALBED</span><span class="o">)</span> <span class="o">{</span>
<span class="n">productRolloutThreadLocal</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="kc">true</span><span class="o">);</span>
<span class="o">}</span> <span class="k">else</span> <span class="o">{</span> <span class="c1">// DRY-RUN case</span>
<span class="n">productRolloutThreadLocal</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="kc">true</span><span class="o">);</span>
<span class="k">try</span> <span class="o">{</span>
<span class="n">delegator</span><span class="o">.</span><span class="na">get</span><span class="o">();</span>
<span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">Exception</span> <span class="n">ex</span><span class="o">)</span> <span class="o">{</span>
<span class="n">log</span><span class="o">.</span><span class="na">warn</span><span class="o">(</span><span class="s">"DRY-RUN has failed: "</span> <span class="o">+</span> <span class="n">ex</span><span class="o">);</span>
<span class="o">}</span>
<span class="n">productRolloutThreadLocal</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="kc">false</span><span class="o">);</span>
<span class="o">}</span>
<span class="k">return</span> <span class="n">delegator</span><span class="o">.</span><span class="na">get</span><span class="o">();</span>
<span class="o">}</span> <span class="k">finally</span> <span class="o">{</span>
<span class="n">productRolloutThreadLocal</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="n">prevFlagValue</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="kd">class</span> <span class="nc">ProductService</span> <span class="o">{</span>
<span class="kd">private</span> <span class="nc">ProductRepository</span> <span class="n">productRepository</span><span class="o">;</span>
<span class="kd">private</span> <span class="nc">FastProductRepository</span> <span class="n">fastProductRepository</span><span class="o">;</span>
<span class="kd">public</span> <span class="nc">Product</span> <span class="nf">getProductDetails</span><span class="o">(</span><span class="nc">Long</span> <span class="n">productId</span><span class="o">)</span> <span class="o">{</span>
<span class="k">if</span> <span class="o">(</span><span class="nc">RolloutTask</span><span class="o">.</span><span class="na">getCurrentFlag</span><span class="o">())</span> <span class="o">{</span>
<span class="k">return</span> <span class="n">fastProductRepository</span><span class="o">.</span><span class="na">find</span><span class="o">(</span><span class="n">productId</span><span class="o">);</span>
<span class="o">}</span>
<span class="k">return</span> <span class="n">productRepository</span><span class="o">.</span><span class="na">find</span><span class="o">(</span><span class="n">productId</span><span class="o">);</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="nc">Product</span> <span class="nf">getProductDetailsFast</span><span class="o">(</span><span class="nc">Long</span> <span class="n">productId</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="n">fastProductRepository</span><span class="o">.</span><span class="na">find</span><span class="o">(</span><span class="n">productId</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Here is how the usage would look like:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">RenderService</span> <span class="o">{</span>
<span class="kd">private</span> <span class="nc">ProductService</span> <span class="n">producService</span><span class="o">;</span>
<span class="kd">private</span> <span class="nc">ProductRolloutService</span> <span class="n">productRolloutService</span><span class="o">;</span>
<span class="kd">public</span> <span class="nc">Page</span> <span class="nf">renderProduct</span><span class="o">(</span><span class="nc">Long</span> <span class="n">productId</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">Product</span> <span class="n">product</span> <span class="o">=</span> <span class="n">productRolloutService</span><span class="o">.</span><span class="na">execute</span><span class="o">(</span><span class="no">FAST_PRODUCT_DISABLED</span><span class="o">,</span> <span class="o">()</span> <span class="o">-></span> <span class="o">{</span>
<span class="n">productService</span><span class="o">.</span><span class="na">getProductDetails</span><span class="o">(</span><span class="n">productId</span><span class="o">);</span>
<span class="o">});</span>
<span class="c1">// Do a lot of other stuff required to render product page</span>
<span class="k">return</span> <span class="nf">renderFinalPage</span><span class="o">(</span><span class="n">product</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>In the above example, the FAST_PRODUCT_DISABLED flag is passed as an argument, indicating that the original DB-read should be used. When the customers of the ProductService are ready, they can begin running in a DRY-Run mode by passing FAST_PRODUCT_DRY_RUN as an argument. The purpose of the dry-run mode is to test the availability of the new fast-DB without fully relying on its data. If all services enable the dry-run mode, it allows us to assess the performance of the new fast-DB under full load. We can operate in this state for a certain period of time, ensuring safety and stability, and once assured, proceed to switch to the FAST_PRODUCT_ENABLED mode.</p>
<h2 id="conclusion">Conclusion</h2>
<p>Advantage of using a Task Runner Wrapper-based rollout is that it allows you to selectively enable the flag for specific calls rather than enabling it for all calls simultaneously. This granular control enables a gradual and controlled rollout of the feature. Additionally, the wrapper can be used to group a few calls under the same scope, providing flexibility in enabling the flag for a specific set of related operations.</p>
<p>Furthermore, this approach facilitates the implementation of a Dry-run mode, allowing for testing and validation without relying on actual data. It also offers the opportunity to measure response times and collect metrics for different versions of the call, aiding in performance evaluation and analysis.</p>
<p>However, it is important to note that this approach requires additional work and careful design to ensure its effectiveness. If not implemented accurately, it could have a negative impact on performance, so it is crucial to design and test the wrapper thoroughly to mitigate any potential performance issues.</p>
<p>As a final piece of advice, it is essential to remember to clean up your feature flags once the experiment or migration is completed. Although it may seem obvious, many developers tend to overlook this step. Removing unnecessary feature flags ensures a clean and organized codebase, avoids confusion, and prevents any unintended consequences that might arise from leaving unused flags in the code. So, always make it a practice to clean up your feature flags when they are no longer needed.</p>Rustam Rakhimov IgorevichIn modern software development, the process extends beyond simply completing a task and moving on. It entails continuous work on new updates, features, and bug fixes. Therefore, the role of feature flags becomes crucial in the software’s lifecycle. Feature flags can be classified as either static or dynamic, depending on the speed at which their changes are applied to the application’s behavior. In this post, we will not discuss the various types of feature flags; instead, we will concentrate on the different approaches for writing code that depends on these variables and styles.Singleton Unit test in typescript2023-01-10T00:00:00+00:002023-01-10T00:00:00+00:00https://rusyasoft.github.io/typescript/2023/01/10/Singleton-UnitTest-in-typescript<p><a href="https://en.wikipedia.org/wiki/Singleton_pattern">Singleton</a> is one of the most popular Design Pattern, that we can meet practically in any project. Sometime we try not to use it and design the project Singleton free, but sooner or later we all end up using it. Then question raise how to unit test it and all languages have methods. I run into a small task, where had to write a unit test for a singleton in typescript. Most of my searches were not that great, where in some places people suggested to have a reset() function, while <a href="https://www.technicalfeeder.com/2020/10/unit-test-for-singleton-class/">others</a> have suggested cheating ways by using ‘any’ casting. Finally got a good suggestion from my colleague to use extend test class from original singleton, and in constructor purposefully call parent’s constructor, where we actually can do proper reset. (credits to <a href="https://www.linkedin.com/in/nathan-magnus">Nathan</a>).</p>
<h2 id="example-of-singleton">Example of Singleton</h2>
<div class="language-typescript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nx">Singleton</span> <span class="p">{</span>
<span class="k">public</span> <span class="k">static</span> <span class="nx">Instance</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">Singleton</span><span class="p">();</span>
<span class="k">private</span> <span class="nx">int</span> <span class="nx">counter</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="k">protected</span> <span class="kd">constructor</span><span class="p">()</span> <span class="p">{</span>
<span class="k">this</span><span class="p">.</span><span class="nx">counter</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">void</span> <span class="nx">increment</span><span class="p">()</span> <span class="p">{</span>
<span class="k">this</span><span class="p">.</span><span class="nx">counter</span><span class="o">++</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">public</span> <span class="nx">int</span> <span class="nx">getCounter</span><span class="p">()</span> <span class="p">{</span>
<span class="k">return</span> <span class="k">this</span><span class="p">.</span><span class="nx">counter</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<h2 id="unit-test-example-of-singleton">Unit Test Example of Singleton</h2>
<p>Here is the test class that would look pretty simple</p>
<div class="language-typescript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nx">SingletonTest</span> <span class="kd">extends</span> <span class="nx">Singleton</span> <span class="p">{</span>
<span class="k">public</span> <span class="kd">constructor</span><span class="p">()</span> <span class="p">{</span>
<span class="k">super</span><span class="p">();</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>After that we can just write our Jest based unit test boilerplate. The following code can be in the same file as SingletonTest, that way you will make sure its not used anywhere else.</p>
<div class="language-typescript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">describe</span><span class="p">(</span><span class="dl">"</span><span class="s2">Singleton unit test</span><span class="dl">"</span><span class="p">,</span> <span class="p">()</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">beforeEach</span><span class="p">(()</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">Singleton</span><span class="p">.</span><span class="nx">Instance</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">SingletonTest</span><span class="p">();</span>
<span class="p">});</span>
<span class="nx">it</span><span class="p">(</span><span class="dl">"</span><span class="s2">lets test Singleton with two increments</span><span class="dl">"</span><span class="p">,</span> <span class="p">()</span> <span class="o">=></span> <span class="p">{</span>
<span class="kd">const</span> <span class="nx">singleton</span> <span class="o">=</span> <span class="nx">Singleton</span><span class="p">.</span><span class="nx">Instance</span><span class="p">;</span>
<span class="nx">singleton</span><span class="p">.</span><span class="nx">increment</span><span class="p">();</span>
<span class="nx">singleton</span><span class="p">.</span><span class="nx">increment</span><span class="p">();</span>
<span class="nx">expect</span><span class="p">(</span><span class="nx">singleton</span><span class="p">.</span><span class="nx">getCounter</span><span class="p">()).</span><span class="nx">toEqual</span><span class="p">(</span><span class="mi">2</span><span class="p">);</span>
<span class="p">});</span>
<span class="nx">it</span><span class="p">(</span><span class="dl">"</span><span class="s2">lets test Singleton with three increments</span><span class="dl">"</span><span class="p">,</span> <span class="p">()</span> <span class="o">=></span> <span class="p">{</span>
<span class="kd">const</span> <span class="nx">singleton</span> <span class="o">=</span> <span class="nx">Singleton</span><span class="p">.</span><span class="nx">Instance</span><span class="p">;</span>
<span class="nx">singleton</span><span class="p">.</span><span class="nx">increment</span><span class="p">();</span>
<span class="nx">singleton</span><span class="p">.</span><span class="nx">increment</span><span class="p">();</span>
<span class="nx">singleton</span><span class="p">.</span><span class="nx">increment</span><span class="p">();</span>
<span class="nx">expect</span><span class="p">(</span><span class="nx">singleton</span><span class="p">.</span><span class="nx">getCounter</span><span class="p">()).</span><span class="nx">toEqual</span><span class="p">(</span><span class="mi">3</span><span class="p">);</span>
<span class="p">});</span>
<span class="p">}</span>
</code></pre></div></div>
<h2 id="conclusion">Conclusion</h2>
<p>Obviously this is very primitive Singleton indeed but it solves main issue. It protects resetting your singleton instance in your main code, unless you will create similar helper class intentionally. We can go more sophisticated way by implementing getter of instance and do creation of instance by condition. If instance is null then create a new again, that way we could avoid using helper test class.</p>
<div class="language-typescript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// assuming we have getInstance() with following impelemntation</span>
<span class="k">public</span> <span class="nx">getInstance</span><span class="p">():</span> <span class="nx">Singleton</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="k">this</span><span class="p">.</span><span class="nx">Instance</span> <span class="o">===</span> <span class="kc">null</span><span class="p">)</span> <span class="p">{</span>
<span class="k">this</span><span class="p">.</span><span class="nx">Instance</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">Singleton</span><span class="p">();</span>
<span class="p">}</span>
<span class="k">return</span> <span class="k">this</span><span class="p">.</span><span class="nx">Instance</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Then resetting would be as easy as follows:</p>
<div class="language-typescript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">beforeEach</span><span class="p">(()</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">Singleton</span><span class="p">.</span><span class="nx">Instance</span> <span class="o">=</span> <span class="kc">null</span><span class="p">;</span>
<span class="p">});</span>
</code></pre></div></div>
<p>then we have to make sure we always going to use getInstance(), nobody can guarantee that by mistake we may use <code class="language-plaintext highlighter-rouge">Singleton.Instance</code>.</p>Rustam Rakhimov IgorevichSingleton is one of the most popular Design Pattern, that we can meet practically in any project. Sometime we try not to use it and design the project Singleton free, but sooner or later we all end up using it. Then question raise how to unit test it and all languages have methods. I run into a small task, where had to write a unit test for a singleton in typescript. Most of my searches were not that great, where in some places people suggested to have a reset() function, while others have suggested cheating ways by using ‘any’ casting. Finally got a good suggestion from my colleague to use extend test class from original singleton, and in constructor purposefully call parent’s constructor, where we actually can do proper reset. (credits to Nathan).Quick Overview of JavaScript Event Loops2022-12-21T00:00:00+00:002022-12-21T00:00:00+00:00https://rusyasoft.github.io/javascript/2022/12/21/Quick-overview-JavaScript-EventLoops<p>This post will be mostly about the quick summary on Event Loops in JavaScript. I think it is crucial thing to understand for JavaScript developers. For the details of please refer to the conference video presented by Erin Zimmer.</p>
<h2 id="event-loop-intro">Event Loop Intro</h2>
<p>Here is the definition from Wikipedia that says “the event loop is a programming construct or design pattern that waits for and dispatches events or messages in a program”. But if we are going to put it more simple its just a forever loop that runs and does something.</p>
<pre><code class="language-JavaScript">while (true) {
doSomething();
}
</code></pre>
<p>In case of javascript it runs the code that lies in the queue.</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">while</span> <span class="p">(</span><span class="kc">true</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">task</span> <span class="o">=</span> <span class="nx">taskQueue</span><span class="p">.</span><span class="nx">pop</span><span class="p">();</span>
<span class="nx">execute</span><span class="p">(</span><span class="nx">task</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Since javascript has mostly two platforms right now: Node-code and browser. The main concept for both quite similar while browser’s case is more complex. Lets look at how the event loop looks like in terms of browser.</p>
<p><img src="/assets/2022/AbstractedBrowserArchitecture.png" alt="Abstract Browser" /></p>
<p>For this post lets just ignore the WEB API’s side and simply lets think of it as an auxiliary functions, that javascript actually delegates to. In that case what we left over is JavaScript engine and Queue (in picture above depicted as Callback-Queue). Things are straightforward, we get the task from the queue execute. While executing we may end up adding new task to the queue and so on. Auxiliary WEB APIs also can generate some tasks and put them into the queue. Most of the time it will be related to external events, like user clicks on something or fetching external api has responded back.</p>
<h2 id="zoomed-in-event-loop">Zoomed In Event Loop</h2>
<p>In reality we have more then a single queue where we store different kinds of tasks. We can categorize them as a Microtasks, Macrotasks, Rendering Pipeline, Animation Frame Callback Queue.</p>
<p>Macrotasks - onclick, fetch and other kind of external events
Microtasks - promises endup in microtasks
Rendering Pipeline - the task when browser actually renders tasks and put things on the screen
Animation Frame Callback Queue - is the queue through which we tell browser that we would like to animate. And we have to request right amount of animation otherwise we may block everything else. <a href="https://developer.mozilla.org/en-US/docs/Web/API/window/requestAnimationFrame">Here</a> is more details about animation frame.</p>
<p>Lets look at the order and conditions of how queues are executed and its best described using following code:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">while</span> <span class="p">(</span><span class="kc">true</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">queue</span> <span class="o">=</span> <span class="nx">getNextQueue</span><span class="p">();</span> <span class="c1">// get next macro-task</span>
<span class="nx">task</span> <span class="o">=</span> <span class="nx">queue</span><span class="p">.</span><span class="nx">pop</span><span class="p">();</span>
<span class="nx">execute</span><span class="p">(</span><span class="nx">task</span><span class="p">);</span>
<span class="k">while</span> <span class="p">(</span><span class="nx">microtaskQueue</span><span class="p">.</span><span class="nx">hasTasks</span><span class="p">())</span> <span class="nx">doMicrotask</span><span class="p">();</span> <span class="c1">// executed all micro-tasks</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">isRepaintTime</span><span class="p">())</span> <span class="p">{</span>
<span class="nx">animationTasks</span> <span class="o">=</span> <span class="nx">animationQueue</span><span class="p">.</span><span class="nx">copyTasks</span><span class="p">();</span> <span class="c1">// fix the number of animations that are in the queue right now, so we run only those</span>
<span class="k">for</span> <span class="p">(</span><span class="nx">task</span> <span class="k">in</span> <span class="nx">animationTasks</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">doAnimationTask</span><span class="p">(</span><span class="nx">task</span><span class="p">);</span> <span class="c1">// animate only fixed number of frames</span>
<span class="p">}</span>
<span class="nx">repaint</span><span class="p">();</span> <span class="c1">// finally run the render-task</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Figure below can be visual representation of these queues.</p>
<p><img src="/assets/2022/QueuesAndOrders.png" alt="QueuesAndOrders" /></p>
<h2 id="summary">Summary</h2>
<p>Rules mentioned in the Erin’s presentation over queues:</p>
<ul>
<li>Queues can be executed in any order.</li>
<li>Tasks in the same queue must be executed in the order they arrived</li>
<li>Tasks from the same source must go in the same queue.</li>
</ul>
<p>If above mentioned principles followed then there should not be any big mess with the order of the running events. But each vendor of browser decides how to optimize the orders and rules over which queue should run when and how often. I’m not exactly sure that every browser would exactly follow the steps/algorithm mentioned above, and it may change in the future as well.</p>
<h2 id="references">References:</h2>
<p><a href="https://www.youtube.com/watch?v=u1kqx6AenYw">Further Adventures of the Event Loop (Erin Zimmer | JSConf EU 2018)</a>
<a href="https://www.youtube.com/watch?v=8aGhZQkoFbQ">What the heck is the event loop anyway? | Philip Roberts | JSConf EU</a></p>Rustam Rakhimov IgorevichThis post will be mostly about the quick summary on Event Loops in JavaScript. I think it is crucial thing to understand for JavaScript developers. For the details of please refer to the conference video presented by Erin Zimmer..Net Core Database Connection Steps2022-08-05T00:00:00+00:002022-08-05T00:00:00+00:00https://rusyasoft.github.io/dotnet/2022/08/05/NetCore-DB-Connection<p>This post describes steps needed to setup connection with .Net Core MVC application and database. May be some of the parts of this code can be used by non-MVC based projects as well (can’t guarantee it because haven’t tried myself). This post has been snippeted from the O’reilly course called <a href="https://learning.oreilly.com/videos/the-complete-guide/9781801074247/">The Complete Guide to ASP.NET Core MVC (.NET 6)</a>.</p>
<p>Here are the rough steps (can be done not particularly in this order):</p>
<ol>
<li>Add Model</li>
<li>Add connection string to appsettings.json</li>
<li>Add ApplicationDbContext class that handles/forwards options of the connection string to DbContext</li>
<li>AddDbContext to Program.cs (Where actual code based initialization is happening)</li>
</ol>
<h2 id="add-model">Add Model</h2>
<p>Under Models folder we create a class which we would like to use in our project. It eventually going to turn into a table in our database.</p>
<pre><code class="language-C#">using System.ComponentModel.DataAnnotations;
namespace BulkyBookWeb.Models;
public class Category
{
[Key]
public int Id { get; set; }
[Required]
public string Name { get; set; }
public int DisplayOrder { get; set; }
public DateTime CreatedDateTime { get; set; } = DateTime.Now;
}
</code></pre>
<p><code class="language-plaintext highlighter-rouge">Id</code> becomes a primary key in our Categories table, since we annotated it with <code class="language-plaintext highlighter-rouge">Key</code>. <code class="language-plaintext highlighter-rouge">Name</code> becomes required column (or non-null column).</p>
<h2 id="add-connection-string">Add Connection String</h2>
<p>Connection string is something that can be declared in many ways, but most best practice is to declare it in a <code class="language-plaintext highlighter-rouge">appsettings.json</code> file. That way we can also distinguish which connection string would be used in a what environment (prod, dev, local etc).</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{
"ConnectionStrings": {
"DefaultConnection": "Server=<localdb-Or-Address-Of-DB-goes-here>;Database=<Database-Name-Intended-to-Connect>;Trusted_Connection=True;"
}
}
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">Trusted_Connection</code> option is telling to application to use Windows Authentication. I don’t think it would be proper to use it in production servers.</p>
<h2 id="creating-a-database">Creating a Database</h2>
<p>In order to store our table, first we need to have our database to be exist. We can go database first model, when we create a database before hand and just use the connection. Another way is called code-first way, where database is created by the code if it doesn’t exist. To interact with database we need db-context, and it is represented as DbContext class. We can create a new ApplicationDbContext class that inherits from DbContext and used to represent our Database.</p>
<p>Note: Microsoft.EntityFrameworkCore package should be installed (by NuGet) before-hand otherwise DbContext class will not be recognized.</p>
<pre><code class="language-C#">using Microsoft.EntityFrameworkCore;
using BulkyBookWeb.Models;
namespace BulkyBookWeb.Data;
public class ApplicationDbContext : DbContext
{
public ApplicationDbContext(DbContextOptions<ApplicationDbContext> options) : base(options)
{
}
public DbSet<Category> Categories { get; set; }
}
</code></pre>
<p>It will create a <code class="language-plaintext highlighter-rouge">Categories</code> table with columns as its defined by Category model.</p>
<h2 id="adding-dbcontext-to-programcs">Adding DbContext to Program.cs</h2>
<p>Following line goes somewhere under <code class="language-plaintext highlighter-rouge">var builder = WebApplication.CreateBuilder(args)</code> but before we actually call <code class="language-plaintext highlighter-rouge">Build()</code> on builder object.</p>
<p>Note: Microsoft.EntityFrameworkCore.SqlServer package should be installed (by NuGet) before-hand, otherwise <code class="language-plaintext highlighter-rouge">UseSqlServer()</code> will not be recognized.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>builder.Services.AddDbContext<ApplicationDbContext>(options => options.UseSqlServer(
builder.Configuration.GetConnectionString("DefaultConnection")
));
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">DefaultConnection</code> is the name that has been chosen at connection string inside <code class="language-plaintext highlighter-rouge">appsettings.json</code> file.</p>
<h2 id="migration">Migration</h2>
<p>In order to actually deploy the database changes, or even creation of new table and columns at the beginning we have to perform migration. I know the wording might be confusing since we are not migrating anything, but that term has been chosen. In order to perform migration we of databases we have to install another NuGet package called <code class="language-plaintext highlighter-rouge">Microsoft.EntityFrameworkCore.Tools</code>.</p>
<p>After that we can go to NuGet package manager console (Tools -> NuGet Package Manager -> Package Manager Console) and run add-migration command.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>PM> add-migration AddCategoryToDatabase
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">AddCategoryToDatabase</code> is just a name of our migration, we can name it whatever we want, but making sensible names would help you in the future. When everything run successfully you get success message.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>PM> add-migration AddCategoryToDatabase
Build started...
Build succeeded.
To undo this action, use Remove-Migration.
</code></pre></div></div>
<p>As a result of this run, under your project you will have a new Migrations folder created. <code class="language-plaintext highlighter-rouge">AddCategoryToDatabase</code> class has two methods <code class="language-plaintext highlighter-rouge">Up</code> (to actually perform changes) and <code class="language-plaintext highlighter-rouge">Down</code> (to rollback changes). Now to perform migrations we should run <code class="language-plaintext highlighter-rouge">update-database</code> command on NuGet package manager console.</p>Rustam Rakhimov IgorevichThis post describes steps needed to setup connection with .Net Core MVC application and database. May be some of the parts of this code can be used by non-MVC based projects as well (can’t guarantee it because haven’t tried myself). This post has been snippeted from the O’reilly course called The Complete Guide to ASP.NET Core MVC (.NET 6).Hello World Application Using HoloLens 22022-07-31T00:00:00+00:002022-07-31T00:00:00+00:00https://rusyasoft.github.io/hololens/2022/07/31/hello-world-with-hololens2<p>Even though its not related to my work directly, I was able to get in touch with Hololens-2. I remember back from my work at KETI (Korea Electronics Technology Institute) when I first time played with Hololens (now its referred as Hololens-1). It was amazing feeling when you can see augmented things with such a great quality and precision just in front of your eye. I would like to admit that Hololens-2 is surely advanced version of previous Hololens. Its much lighter, faster, comfortably sits on your head, can fold screen to your top-head when you stop interacting with mixed-reality.</p>
<p>When I wrote my first application on Hololens-1 it was kinda easy and steps were straight forward. Its been long time since then, so I don’t exactly recall the steps. But overall steps are pretty similar for hololens-2. You setup Unity, then setup plugin for Hololens, then create a Hololens project put 3D stuff you like in it, make a build. Build in Unity will generate Visual Studio project, which you can open separately and compile, and finally deploy on device.</p>
<h2 id="mrtk-tooling">MRTK Tooling</h2>
<p>As I have mentioned above, steps to build Hololens-2 kinda got complicated, compared to what I have experienced before for Hololens-1 build setup. Because now there is a tooling called <a href="https://docs.microsoft.com/en-us/windows/mixed-reality/mrtk-unity/mrtk2/?view=mrtkunity-2022-05">MRTK</a> is introduced. Again I don’t exactly remember using this tool before, but it was kinda little confusing at the setup time. But eventually it worked, so may be little improvement of documentation could help.</p>
<p>Other than that Steps for build setup are mostly same except Unity project relies on setups/configurations from MRTK tool. May be I don’t see the whole advantage of actually having this MRTK tooling because, I was just aiming for “Hello World” application. Because I can see it has very advanced settings/configurations that can affect your build.</p>
<h2 id="tutorial-links-for-quick-setup">Tutorial Links For Quick Setup</h2>
<p>For people whoever wants to do quick getting started, <a href="https://docs.microsoft.com/en-us/learn/paths/beginner-hololens-2-tutorials/">this</a> is the official tutorial, so feel free to follow it. You will end-up creating a two boxes in your Mixed-reality application. I did add “Hello World” text by using <a href="https://learn.unity.com/tutorial/working-with-textmesh-pro#">textmesh-pro</a> and added some non-functioning buttons.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/QE4DaXttXic" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>Rustam Rakhimov IgorevichEven though its not related to my work directly, I was able to get in touch with Hololens-2. I remember back from my work at KETI (Korea Electronics Technology Institute) when I first time played with Hololens (now its referred as Hololens-1). It was amazing feeling when you can see augmented things with such a great quality and precision just in front of your eye. I would like to admit that Hololens-2 is surely advanced version of previous Hololens. Its much lighter, faster, comfortably sits on your head, can fold screen to your top-head when you stop interacting with mixed-reality.Serving aws lambda frugally via ALB(Application Load Balancer)2021-12-21T00:00:00+00:002021-12-21T00:00:00+00:00https://rusyasoft.github.io/expressjs,%20nodejs,%20aws,%20lambda,%20frugality,%20alb/2021/12/21/serving-lambda-frugally-via-alb<p>If we have a goal to build a service using lambda we can serve it using <a href="https://aws.amazon.com/api-gateway/">API-Gateway</a> which is fully managed service from aws that makes it easy for developers to create, publish, maintain, monitor and secure of API at any scale. We can find many good examples over there for this kind of stack, but in this post I would like to talk about serving lambda via ALB. Which could endup being cheaper or even free (if you use free 12 month tier aws account). I don’t want to dive into the discussion of what is better API-Gateway where you pay per request count or ALB which is payed by running hours. One thing that challenged me was to setup web-app that would be served as an ALB –> Lambda combined stack.</p>
<h2 id="alb-to-lambda-setup-via-console">ALB to Lambda setup via console</h2>
<p>In order to make it work we should know that ALB should be backed up by Target Group. Target Group is aws resource that decides where the actual calls are forwarded. Forwarding can be done to IP based, instance based or lambda based resources as an entities of the target group.</p>
<p><img src="/assets/2021/12/alb-images/alb-listener.png" alt="ALB Listener" /></p>
<p>ALB listener has a rule where port 80 is listened for http incoming requests and forwarded to target-group (named <code class="language-plaintext highlighter-rouge">Hamma-LBHTT-52KRLLFSW2ON</code>). It is also possible to setup https end-point by attaching SSL certificates to ALB listener. In above example we don’t have any specific rules regarding of forwarding incoming packages, but if it is required ALB has a quite good <a href="https://docs.aws.amazon.com/elasticloadbalancing/latest/application/listener-update-rules.html">rule engine</a>.</p>
<h3 id="vpc-based-lambda">VPC based Lambda</h3>
<p>The citation from aws official site says:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>By default, Lambda runs your functions in a secure VPC with access to AWS services and the internet. Lambda owns this VPC, which isn't connected to your account's default VPC. When you connect a function to a VPC in your account, the function can't access the internet unless your VPC provides access.
</code></pre></div></div>
<p>The wording may sound little confsing, but lets try to put it in a more plain English. By default lambda is not attached to our account’s VPC, so if we want to interact with in VPC resources (Redis, RDS, EC2 etc) then we have to put lambda into our VPC. Also since lambda by default is not attached to our VPC it has no access to global internet. In summary, you get lambda which has no connection to internet and can only receive incoming requests from ALB. For any application that does something useful in most cases we need at least internet connection.</p>
<p><img src="/assets/2021/12/alb-images/vpc-settings-of-lambda.png" alt="VPC-settings-of-lambda" /></p>
<p>VPC for lambda can be set at “lambda -> Configuration -> VPC”. Another important thing to note is that lambda should be put under private subnet of the VPC, puting it under public subnet is not gonna work. Obviously if you want to allow the flow from ALB to port 80 then Security-group of the lambda should have port 80 in allow list of inboud rules.</p>
<h2 id="alb-to-lambda-setup-via-cdk-iac">ALB to Lambda setup via CDK (IaC)</h2>
<p>All above setting can be also done using CDK, following code snippets can be served as a good example for starters:</p>
<div class="language-typescript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// Create a VPC in two AZ</span>
<span class="kd">const</span> <span class="nx">vpc</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">ec2</span><span class="p">.</span><span class="nx">Vpc</span><span class="p">(</span><span class="k">this</span><span class="p">,</span> <span class="dl">"</span><span class="s2">VPC</span><span class="dl">"</span><span class="p">,</span> <span class="p">{</span>
<span class="na">natGateways</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="na">maxAzs</span><span class="p">:</span> <span class="mi">2</span>
<span class="p">});</span>
<span class="c1">// Lambda function declaration</span>
<span class="kd">const</span> <span class="nx">lambdaFunction</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">lambda</span><span class="p">.</span><span class="nb">Function</span><span class="p">(</span><span class="k">this</span><span class="p">,</span> <span class="dl">'</span><span class="s1">lambda-function</span><span class="dl">'</span><span class="p">,</span> <span class="p">{</span>
<span class="na">runtime</span><span class="p">:</span> <span class="nx">lambda</span><span class="p">.</span><span class="nx">Runtime</span><span class="p">.</span><span class="nx">NODEJS_14_X</span><span class="p">,</span>
<span class="na">memorySize</span><span class="p">:</span> <span class="mi">1024</span><span class="p">,</span>
<span class="na">timeout</span><span class="p">:</span> <span class="nx">cdk</span><span class="p">.</span><span class="nx">Duration</span><span class="p">.</span><span class="nx">seconds</span><span class="p">(</span><span class="mi">60</span><span class="p">),</span>
<span class="na">handler</span><span class="p">:</span> <span class="dl">'</span><span class="s1">lambda.handler</span><span class="dl">'</span><span class="p">,</span>
<span class="na">code</span><span class="p">:</span> <span class="nx">lambda</span><span class="p">.</span><span class="nx">Code</span><span class="p">.</span><span class="nx">fromAsset</span><span class="p">(</span><span class="nx">path</span><span class="p">.</span><span class="nx">join</span><span class="p">(</span><span class="nx">__dirname</span><span class="p">,</span> <span class="dl">'</span><span class="s1">/path/to/the/asset/folder/</span><span class="dl">'</span><span class="p">)),</span>
<span class="na">environment</span><span class="p">:</span> <span class="p">{</span>
<span class="na">REGION</span><span class="p">:</span> <span class="nx">cdk</span><span class="p">.</span><span class="nx">Stack</span><span class="p">.</span><span class="k">of</span><span class="p">(</span><span class="k">this</span><span class="p">).</span><span class="nx">region</span><span class="p">,</span>
<span class="na">AVAILABILITY_ZONES</span><span class="p">:</span> <span class="nx">JSON</span><span class="p">.</span><span class="nx">stringify</span><span class="p">(</span>
<span class="nx">cdk</span><span class="p">.</span><span class="nx">Stack</span><span class="p">.</span><span class="k">of</span><span class="p">(</span><span class="k">this</span><span class="p">).</span><span class="nx">availabilityZones</span><span class="p">,</span>
<span class="p">),</span>
<span class="p">},</span>
<span class="p">});</span>
<span class="c1">// Create an internet-facing Application Load Balancer using the VPC</span>
<span class="kd">const</span> <span class="nx">lb</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">elbv2</span><span class="p">.</span><span class="nx">ApplicationLoadBalancer</span><span class="p">(</span><span class="k">this</span><span class="p">,</span> <span class="dl">"</span><span class="s2">LB</span><span class="dl">"</span><span class="p">,</span> <span class="p">{</span>
<span class="nx">vpc</span><span class="p">,</span>
<span class="na">internetFacing</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span>
<span class="p">});</span>
<span class="c1">// Add a listener on port 443 for and use the certificate for HTTPS</span>
<span class="kd">const</span> <span class="nx">listenerHTTP</span> <span class="o">=</span> <span class="nx">lb</span><span class="p">.</span><span class="nx">addListener</span><span class="p">(</span><span class="dl">"</span><span class="s2">HTTPSListener</span><span class="dl">"</span><span class="p">,</span> <span class="p">{</span>
<span class="na">port</span><span class="p">:</span> <span class="mi">80</span><span class="p">,</span>
<span class="p">});</span>
<span class="c1">// Add a target with the AWS Lambda function to the listener</span>
<span class="nx">listenerHTTP</span><span class="p">.</span><span class="nx">addTargets</span><span class="p">(</span><span class="dl">"</span><span class="s2">HTTPSListenerTargets</span><span class="dl">"</span><span class="p">,</span> <span class="p">{</span>
<span class="na">targets</span><span class="p">:</span> <span class="p">[</span><span class="k">new</span> <span class="nx">targets</span><span class="p">.</span><span class="nx">LambdaTarget</span><span class="p">(</span><span class="nx">lambdaFunction</span><span class="p">)],</span>
<span class="na">healthCheck</span><span class="p">:</span> <span class="p">{</span>
<span class="na">enabled</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span>
<span class="p">},</span>
<span class="p">});</span>
<span class="c1">// Printing out ALB Entry point</span>
<span class="k">new</span> <span class="nx">cdk</span><span class="p">.</span><span class="nx">CfnOutput</span><span class="p">(</span><span class="k">this</span><span class="p">,</span> <span class="dl">'</span><span class="s1">ALB Entry point</span><span class="dl">'</span><span class="p">,</span> <span class="p">{</span>
<span class="na">value</span><span class="p">:</span> <span class="nx">lb</span><span class="p">.</span><span class="nx">loadBalancerDnsName</span>
<span class="p">});</span>
</code></pre></div></div>
<p>The code above is self descriptive, only thing to mention here is we need to have NAT resource connected to our private subnet. Otherwise private and public subnetworks would have no connection and private subnet would have no internet connection. There are two kinds of NAT solutions are avaialbe in <a href="https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat-comparison.html">aws</a>: NAT-Gateway service and NAT-Instances.</p>
<p>Following is the example of creating VPC with either of NAT kind based on switch flag:</p>
<div class="language-typescript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">export</span> <span class="kr">enum</span> <span class="nx">NatType</span> <span class="p">{</span>
<span class="nx">GATEWAY</span> <span class="o">=</span> <span class="dl">"</span><span class="s2">NatGateway</span><span class="dl">"</span><span class="p">,</span>
<span class="nx">INSTANCE</span> <span class="o">=</span> <span class="dl">"</span><span class="s2">NatInstance</span><span class="dl">"</span>
<span class="p">}</span>
<span class="k">private</span> <span class="nx">createVPC</span><span class="p">(</span><span class="nx">natType</span><span class="p">:</span> <span class="nx">NatType</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">const</span> <span class="nx">vpcProps</span> <span class="o">=</span> <span class="p">(</span><span class="nx">natType</span> <span class="o">==</span> <span class="nx">NatType</span><span class="p">.</span><span class="nx">INSTANCE</span><span class="p">)?</span>
<span class="p">{</span>
<span class="na">maxAzs</span><span class="p">:</span> <span class="mi">2</span><span class="p">,</span>
<span class="na">natGatewayProvider</span><span class="p">:</span> <span class="k">new</span> <span class="nx">ec2</span><span class="p">.</span><span class="nx">NatInstanceProvider</span><span class="p">({</span>
<span class="na">instanceType</span><span class="p">:</span> <span class="k">new</span> <span class="nx">ec2</span><span class="p">.</span><span class="nx">InstanceType</span><span class="p">(</span><span class="dl">'</span><span class="s1">t2.nano</span><span class="dl">'</span><span class="p">),</span>
<span class="p">})</span>
<span class="p">}</span>
<span class="p">:</span>
<span class="p">{</span>
<span class="na">natGateways</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="na">maxAzs</span><span class="p">:</span> <span class="mi">2</span><span class="p">,</span>
<span class="p">}</span>
<span class="k">return</span> <span class="k">new</span> <span class="nx">ec2</span><span class="p">.</span><span class="nx">Vpc</span><span class="p">(</span><span class="k">this</span><span class="p">,</span> <span class="dl">"</span><span class="s2">VPC</span><span class="dl">"</span><span class="p">,</span> <span class="nx">vpcProps</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<h2 id="lambda-quota-limits">Lambda Quota Limits</h2>
<p>If you are going for a production with this setup then don’t forget to request quota extension of for concurrency limit. By default it is 1000 which is quite a big enough. But if you are considering big load it is something you should look into this. Requesting limit increase doesn’t require any additional charges from you. So called soft limit 1000 can be easily extended to 10’000 (tried for Frankfurt region). But if you need even more than that, then you may need to explain details of your application and load pattern to the AWS Customer Service. No worries, if you need it they will provide it (at the end you pay for what you use anyway)</p>
<h2 id="what-nosql-to-use-">What NoSQL to use ?</h2>
<p>Since lambda is the serverless compute it would be more proper to use serverless DB accordingly (dynamodb). But in most cases when we migrate to AWS we want to reuse the DB which we already have expertise on. MongoDB worked well for the beginning, but during stress-test it was the bottleneck. Our application start failing because MongoDB couldn’t handle many number of connections. Even declaring the connection globally and reusing existing connection is still wasn’t enough. The reason is we don’t control which lambda contain instances lives and which one whipes out. When the container whiped out, we start having zombie mongoDB connections. In my stress test we try to run mongoDB on premise and ec2 the results were same. Next step was adding a redish cache layer which did allow us to increase number of incoming concurrent request towards the lambda.</p>Rustam Rakhimov IgorevichIf we have a goal to build a service using lambda we can serve it using API-Gateway which is fully managed service from aws that makes it easy for developers to create, publish, maintain, monitor and secure of API at any scale. We can find many good examples over there for this kind of stack, but in this post I would like to talk about serving lambda via ALB. Which could endup being cheaper or even free (if you use free 12 month tier aws account). I don’t want to dive into the discussion of what is better API-Gateway where you pay per request count or ALB which is payed by running hours. One thing that challenged me was to setup web-app that would be served as an ALB –> Lambda combined stack.ExpressJS as AWS Lambda (Frugality)2021-12-20T00:00:00+00:002021-12-20T00:00:00+00:00https://rusyasoft.github.io/expressjs,%20nodejs,%20aws,%20lambda,%20frugality/2021/12/20/expressjs-in-lambda<p>Running expressJS in lambda we can actually build very frugal startup project, which can have verymuch unknown and unpredictable outcome. That feeling when you have built something and invested money for infrastructure but no user show up. In order to prevent that kind of frustrating experience we could actually start the project with lambda. Which could save us some cash.</p>
<p>ExpressJS can be quickly started by following <a href="https://expressjs.com/en/starter/generator.html">official</a> documentation, or refer google for overwhelming information to get started.</p>
<h2 id="concept">Concept</h2>
<p>Before diving into the code lets do some quick analysis. If we follow standard way of developing expressjs application then we endup having following directory structure:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>.
├── app.js
├── bin
│ └── www
├── package.json
├── public
│ ├── images
│ ├── javascripts
│ └── stylesheets
│ └── style.css
├── routes
│ ├── index.js
│ └── users.js
└── views
├── error.pug
├── index.pug
└── layout.pug
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">app.js</code> is the entry point to our server application. As we know most of the codes are written at <code class="language-plaintext highlighter-rouge">routes</code> and related sections. Lets try to keep our <code class="language-plaintext highlighter-rouge">app.js</code> as independent and clean possible from actual business logics.</p>
<p>Another side: it is well known tha aws lambda also has a entry point which is usually handler function in following form:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>exports.handler = (event, context) => {
...
}
</code></pre></div></div>
<p>Now we should have some way to bind these two entry points and make our expressJS work under lambda. <a href="https://www.npmjs.com/package/aws-serverless-express">AWS-Serverless-Express</a> module is the who allows us to wrap our express app to to serverless use. There are some other libraries also available if you are interested, but I personally decided to use that one. Because I noticed its actively <a href="https://github.com/vendia/serverless-express">contirbuted</a></p>
<h2 id="wrapping-express-into-serverless">Wrapping Express into Serverless</h2>
<p>Wrapping express into serverless would have following look:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="dl">'</span><span class="s1">use strict</span><span class="dl">'</span>
<span class="kd">const</span> <span class="nx">awsServerlessExpress</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="dl">'</span><span class="s1">aws-serverless-express</span><span class="dl">'</span><span class="p">)</span>
<span class="kd">const</span> <span class="nx">app</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="dl">'</span><span class="s1">./app</span><span class="dl">'</span><span class="p">)</span>
<span class="kd">const</span> <span class="nx">server</span> <span class="o">=</span> <span class="nx">awsServerlessExpress</span><span class="p">.</span><span class="nx">createServer</span><span class="p">(</span><span class="nx">app</span><span class="p">)</span>
<span class="nx">exports</span><span class="p">.</span><span class="nx">handler</span> <span class="o">=</span> <span class="p">(</span><span class="nx">event</span><span class="p">,</span> <span class="nx">context</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span> <span class="nx">awsServerlessExpress</span><span class="p">.</span><span class="nx">proxy</span><span class="p">(</span><span class="nx">server</span><span class="p">,</span> <span class="nx">event</span><span class="p">,</span> <span class="nx">context</span><span class="p">)</span> <span class="p">}</span>
</code></pre></div></div>
<p>If you want to serve not only apis but also other other pages from your lambda then mime types must be added. It can be done as follows:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="dl">'</span><span class="s1">use strict</span><span class="dl">'</span>
<span class="kd">const</span> <span class="nx">awsServerlessExpress</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="dl">'</span><span class="s1">aws-serverless-express</span><span class="dl">'</span><span class="p">)</span>
<span class="kd">const</span> <span class="nx">app</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="dl">'</span><span class="s1">./app</span><span class="dl">'</span><span class="p">)</span>
<span class="kd">const</span> <span class="nx">binaryMimeTypes</span> <span class="o">=</span> <span class="p">[</span>
<span class="dl">'</span><span class="s1">application/javascript</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">application/json</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">application/octet-stream</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">application/xml</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">font/eot</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">font/opentype</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">font/otf</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">image/*</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">text/comma-separated-values</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">text/css</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">text/html</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">text/javascript</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">text/plain</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">text/text</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">text/xml</span><span class="dl">'</span>
<span class="p">];</span>
<span class="kd">const</span> <span class="nx">server</span> <span class="o">=</span> <span class="nx">awsServerlessExpress</span><span class="p">.</span><span class="nx">createServer</span><span class="p">(</span><span class="nx">app</span><span class="p">,</span> <span class="kc">null</span><span class="p">,</span> <span class="nx">binaryMimeTypes</span><span class="p">)</span>
<span class="nx">exports</span><span class="p">.</span><span class="nx">handler</span> <span class="o">=</span> <span class="p">(</span><span class="nx">event</span><span class="p">,</span> <span class="nx">context</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span> <span class="nx">awsServerlessExpress</span><span class="p">.</span><span class="nx">proxy</span><span class="p">(</span><span class="nx">server</span><span class="p">,</span> <span class="nx">event</span><span class="p">,</span> <span class="nx">context</span><span class="p">)</span> <span class="p">}</span>
</code></pre></div></div>
<p>The code snippet shown above can be stored into <code class="language-plaintext highlighter-rouge">lambda.js</code> file and put at the same folder as <code class="language-plaintext highlighter-rouge">app.js</code>. To bring the dependecy add <code class="language-plaintext highlighter-rouge">"aws-serverless-express": "^3.4.0"</code> (version may change in the future) to the dependencies section of <code class="language-plaintext highlighter-rouge">package.json</code> file.</p>
<h2 id="conclusion">Conclusion</h2>
<p>It may be considered as not efficient solution since we have additional overhead operation (loading a framework) everytime when the request comes in. Since this is a nodejs code and it runs much faster than other languages on top of lambda, it is bearable. Indeed this solution is for people who already have their setup on ExpressJS and lazy to re-design architecture for serverless specific. If you are not lazy and ready to take a challenge on serverless then its good to re-design by SPA <–> API(Serverless) architecture.</p>Rustam Rakhimov IgorevichRunning expressJS in lambda we can actually build very frugal startup project, which can have verymuch unknown and unpredictable outcome. That feeling when you have built something and invested money for infrastructure but no user show up. In order to prevent that kind of frustrating experience we could actually start the project with lambda. Which could save us some cash.How Browser Internal Works (Event Manager)2021-12-19T00:00:00+00:002021-12-19T00:00:00+00:00https://rusyasoft.github.io/browser,%20html,%20javascript,%20dom/2021/12/19/how-browser-internals-work<p>This short blog can be used as a quick cheat-sheet for understanding how Web-browser processes the pages. In general everything in the page should be divided and considered under three sections: DOM objects, CSS objects and JS objects. While browser runs through the page it builds a tree out of objects which is later given to renderer to actually draw a page for user.</p>
<p>To make it easy to imagine following is the figure of that tree (represented in JSON format):</p>
<p><img src="/assets/2021/12/web-browser/tree-objects.png" alt="Browser tree objects" /></p>
<p>Description of sections of document object:</p>
<ul>
<li><strong>DOM TREE</strong>: is the tree that contains pure html objects to be drawn</li>
<li><strong>CSSOM TREE</strong>: is the tree that contains CSS objects that are read either from CSS file or in-code CSS tags</li>
<li><strong>DOM API</strong>: is the javaScript functions provided by default, using which we can access DOM-Tree objects. If we add more custom javaScript codes, you can imagine them ending up in this section</li>
</ul>
<h2 id="event-manager">Event Manager</h2>
<p>Another important concept to know in browser is the Event Manager. Event Manager is responsible to interactive part of the browser, where browser interacts with Operating System by getting ingested various events. Operating System events can be:</p>
<ul>
<li>Time change event</li>
<li>Browser resive event</li>
<li>Internet (network incoming) event</li>
<li>Keyboard and mouse event</li>
</ul>
<p><img src="/assets/2021/12/web-browser/event-manager-in-action.png" alt="Event Manager in Action" /></p>
<p>Event-Manager listens for events from OS, when it receives new event it checks Event-Registry table and creates EventObjects only if JS interested in that event. Event-Registry table is filled out when the page is processed first time, if the Document tree object contains some callback event used, then it is registered in Event-Registry table. That way javaScript compiler can perform manipulative actions on DOM/BOM objects whenever external events come in.</p>
<p>Not so clear? lets go step by step explanation of the full cycle of registering and calling events.</p>
<p><img src="/assets/2021/12/web-browser/registering-callback-function-into-event-registry.png" alt="Event Manager in Action" /></p>
<p>Figure above explains, when compiler sees onclick action assigns showWelcome() function it registers it to Events-Registry table asynchronously.</p>
<p><img src="/assets/2021/12/web-browser/mouseclick-action-interact-with-event-registry.png" alt="Event Manager in Action" /></p>
<p>Later when mouse-click action happens it checks the Events-Registry table and sees there is a function to call for mouse click action and goes for triggering showWelcome() function.</p>
<p>There are many kind of JS event objects can be registered in the Event-Registry table, some of them are: MouseEvent, KeyboardEvent, XMLHttpRequestEvent(Ajax) etc.</p>Rustam Rakhimov IgorevichThis short blog can be used as a quick cheat-sheet for understanding how Web-browser processes the pages. In general everything in the page should be divided and considered under three sections: DOM objects, CSS objects and JS objects. While browser runs through the page it builds a tree out of objects which is later given to renderer to actually draw a page for user.Read Parquet File using S3 Select2021-11-11T00:00:00+00:002021-11-11T00:00:00+00:00https://rusyasoft.github.io/aws,%20s3,%20storage/2021/11/11/s3-select<p>Parquet format is quite popular and well-used among big-data engineers, and most of the case they have setup to read and check the content of parquet files. But sometimes we may need to quickly check the content of parquet. There are many ways and tools to do that, but I would like to talk about the way to do that using S3 Select feature. Most people who has taken any aws course most probably heard about it, but it passes quickly and unnoticably to our ears. It is quite powerful quick tool to know.</p>
<h2 id="quick-overview-on-s3-select-and-glacier-select">Quick Overview on S3 Select and Glacier Select</h2>
<p>S3 Select is a great feature when you quickly need to query your data stored on S3. Without spending any time to ramp-up the cluster that could do the same job. Lets see the quick definitions of these features:</p>
<ul>
<li>S3 Select: uses SQL statements to filter out the contents of S3 objects and retrieve just the subset of required data</li>
<li>Glacier Select: uses SQL statements directly on data in S3 Glacier without having to restore data to a more frequently accessible tier</li>
</ul>
<p>Below we can see quick aws services comparison table:</p>
<p><img src="/assets/2021/11/S3SelectComparisonTable.png" alt="Comparison with Other Services" /></p>
<p>Important to remember S3 Select is used when query is performed on single S3 object. It can return just a few bytes in a large object. Another advantage is that query runs on S3 itself rather than on your application. If the query need to be run over all S3 bucket objects then we should consider other services (Athena could be a good candidate)</p>
<h2 id="query-parquet-file-in-s3">Query Parquet file in S3</h2>
<p>Below is a quick python lambda-example that does select inside the S3, parquet format:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">json</span>
<span class="kn">import</span> <span class="nn">boto3</span>
<span class="n">s3</span> <span class="o">=</span> <span class="n">boto3</span><span class="p">.</span><span class="n">client</span><span class="p">(</span><span class="s">'s3'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">lambda_handler</span><span class="p">(</span><span class="n">event</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
<span class="c1"># define bucket and object-key
</span> <span class="n">bucket</span> <span class="o">=</span> <span class="n">event</span><span class="p">[</span><span class="s">'bucket'</span><span class="p">]</span>
<span class="n">key</span> <span class="o">=</span> <span class="n">event</span><span class="p">[</span><span class="s">'key'</span><span class="p">]</span>
<span class="c1"># just putting some limit for a querying
</span> <span class="n">limit</span> <span class="o">=</span> <span class="mi">10</span>
<span class="k">if</span> <span class="p">(</span><span class="s">'limit'</span> <span class="ow">in</span> <span class="n">event</span><span class="p">):</span>
<span class="n">limit</span> <span class="o">=</span> <span class="n">event</span><span class="p">[</span><span class="s">'limit'</span><span class="p">]</span>
<span class="n">r</span> <span class="o">=</span> <span class="n">s3</span><span class="p">.</span><span class="n">select_object_content</span><span class="p">(</span>
<span class="n">Bucket</span><span class="o">=</span><span class="n">bucket</span><span class="p">,</span>
<span class="n">Key</span><span class="o">=</span><span class="n">key</span><span class="p">,</span>
<span class="n">ExpressionType</span><span class="o">=</span><span class="s">'SQL'</span><span class="p">,</span>
<span class="n">Expression</span><span class="o">=</span><span class="s">"select * from s3object limit "</span> <span class="o">+</span> <span class="nb">str</span><span class="p">(</span><span class="n">limit</span><span class="p">),</span>
<span class="n">InputSerialization</span><span class="o">=</span><span class="p">{</span><span class="s">'Parquet'</span><span class="p">:</span> <span class="p">{}},</span>
<span class="n">OutputSerialization</span> <span class="o">=</span> <span class="p">{</span><span class="s">'CSV'</span><span class="p">:</span> <span class="p">{}},</span>
<span class="p">)</span>
<span class="n">lines</span> <span class="o">=</span> <span class="p">[]</span>
<span class="c1"># iterate through a results
</span> <span class="k">for</span> <span class="n">event</span> <span class="ow">in</span> <span class="n">r</span><span class="p">[</span><span class="s">'Payload'</span><span class="p">]:</span>
<span class="k">if</span> <span class="s">'Records'</span> <span class="ow">in</span> <span class="n">event</span><span class="p">:</span>
<span class="n">records</span> <span class="o">=</span> <span class="n">event</span><span class="p">[</span><span class="s">'Records'</span><span class="p">][</span><span class="s">'Payload'</span><span class="p">].</span><span class="n">decode</span><span class="p">(</span><span class="s">'utf-8'</span><span class="p">)</span>
<span class="n">lines</span><span class="p">.</span><span class="n">append</span><span class="p">(</span> <span class="n">records</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="n">records</span><span class="p">)</span>
<span class="k">elif</span> <span class="s">'Stats'</span> <span class="ow">in</span> <span class="n">event</span><span class="p">:</span>
<span class="n">statsDetails</span> <span class="o">=</span> <span class="n">event</span><span class="p">[</span><span class="s">'Stats'</span><span class="p">][</span><span class="s">'Details'</span><span class="p">]</span>
<span class="k">print</span><span class="p">(</span><span class="s">"Stats details bytesScanned: "</span><span class="p">)</span>
<span class="k">return</span> <span class="p">{</span>
<span class="s">'statusCode'</span><span class="p">:</span> <span class="mi">200</span><span class="p">,</span>
<span class="s">'body'</span><span class="p">:</span> <span class="n">lines</span>
<span class="p">}</span>
</code></pre></div></div>
<p>In order to run this lambda you can give following test input json object:</p>
<pre><code class="language-JSON">{
"bucket": "rustams-bucket-or-any-bucket-you-want",
"key": "here/you/can/define/path/to/parquetobject/part-00000-e04839d0-c611-4613-b4e7-aa8ae7a4cba3-c000.snappy.parquet"
}
</code></pre>
<h2 id="conclusion">Conclusion</h2>
<p>As we can see it is quick tool to look into a single S3 object content and perform SQL query inside. We can easily extend this application that could scan through a S3 bucket and perform queries individually in each object. But then we have to build a some kind of reduce model to collect all responses into one. That is why when it is needed to query over a bucket Athena is a more preferred service.</p>Rustam Rakhimov IgorevichParquet format is quite popular and well-used among big-data engineers, and most of the case they have setup to read and check the content of parquet files. But sometimes we may need to quickly check the content of parquet. There are many ways and tools to do that, but I would like to talk about the way to do that using S3 Select feature. Most people who has taken any aws course most probably heard about it, but it passes quickly and unnoticably to our ears. It is quite powerful quick tool to know.