Thursday, February 4, 2021

AWS Big Data related notes


DMS 

EndPoint - Connection error for S3 source endpoint - Test Endpoint failed: Application-Status: 1020912, Application-Message: Failed to connect to database.

Root Cause: DMS Replication Instance and the S3 bucket were not in the same region.

          Migration process - 

1) create source & target endpoints and test the connection

2) create migration task using the endpoints (map columns as required) and run it. 

https://www.youtube.com/watch?v=_wb-IupX9JU 

 https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.S3.html

ignoreHeaderRows=1 to ignore header in data

Load Task completed - but no data loaded >> look at the "Bucket folder" value provided in the endpoint --> it should point until the <schema>/<table>/data.csv path.

So, if the sub dirs for schema & table are directly under the bucket - keep it empty 

https://aws.amazon.com/premiumsupport/knowledge-center/dms-task-successful-no-data-s3/  


Taxi_trips

https://www.kaggle.com/divineunited/exploring-the-chicago-taxi-trip-dataset 

https://data.cityofchicago.org/Transportation/Taxi-Trips/wrvz-psew/data 

Data types etchttps://docs.aws.amazon.com/dms/latest/userguide/dms-ug.pdf

CloudWatch logs not written - says Log Group not available or log stream not available - why ?

For DMS to write CloudWatch logs - there should be a Role named  "dms-cloudwatch-logs-role" with "AmazonDMSCloudWatchLogsRole" policy attached to it. If its not there, create manually

Refer: https://aws.amazon.com/premiumsupport/knowledge-center/dms-cloudwatch-logs-not-appearing/ 

Important Note: 

"ColumnNullable" is true by default even if not mentioned in the Schema JSON. Which means, if any column value is empty/null, job will fail at that point.

Data is loaded to RDS one by one (committed one by one sequnetially) - So even if job fails on data error on a particular row, all rows till then will be loaded, and remaining will be skipped.

So, analyze the data before the load - to mark all nullable columns with schema attribute 

"ColumnNullable":"true"

Replication Instance, EndPoints & Replication Tasks


 


 

S3 Data Dir Structure 

  1. Amazon S3
  2. srees-data-bucket-2
  3. publicdata/


  4.  S3 Data to load to RDS (MySQL)

EndPoint settings

Extra connection attributes : bucketName=srees-data-bucket-2;cdcPath=undefined;compressionType=NONE;csvDelimiter=,;csvRowDelimiter=\n;datePartitionEnabled=false;ignoreHeaderRows=1; 

 Table structure (JSON)

{
    "TableCount":"1",
    "Tables": [
        {
            "TableName":"taxi_trips",
            "TablePath":"publicdata/taxi_trips_202007/",
            "TableOwner":"publicdata",
            "TableColumns": [
                {
                    "ColumnName":"trip_id",
                    "ColumnType":"STRING",
                    "ColumnNullable":"false",
                    "ColumnIsPk":"true",
                    "ColumnLength":"50"
                },{
                    "ColumnName":"taxi_id",
                    "ColumnType":"STRING",
                    "ColumnNullable":"false",
                    "ColumnIsPk":"false",
                    "ColumnLength":"150"
                }, ................................................... 
                {
                    "ColumnName":"trip_miles",
                    "ColumnType":"NUMERIC",
                    "ColumnPrecision":"6", --> TOTAL SIZE
                    "ColumnScale":"2", --> #DECIMAL PLACES
                    "ColumnNullable":"true"
                }, 
 .....................................................
                {   "ColumnName":"dropoff_location",
                    "ColumnType":"STRING",
                    "ColumnLength":"30",
                    "ColumnNullable":"true"
                }
            ],
            "TableColumnsTotal":"23"
        }
    ]

Cloud Watch Logs - sample


 

 



 

 

 

AWS API Gateway

Issue faced while invoking it via JQuery:

Access to XMLHttpRequest at 'https://dummyCode.execute-api.us-west-1.amazonaws.com/dev/contactMeToCallSES' from origin 'null' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource. 

Enable it in API Gateway - 

Choose Enable CORS from the Actions drop-down menu.

CORS is required to call your API from a webpage that isn’t hosted on the same domain. To enable CORS for a REST API, set the Access-Control-Allow-Origin header in the response object that you return from your function code.

Access to XMLHttpRequest at 'https://2491zdgh7a.execute-api.us-west-1.amazonaws.com/dev/contactMeToCallSES' from origin 'null' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.

*** Make sure to deploy the resource after this change


***** Problem I faced with APIGateway-Lambda integration - requests were failing *****
API Test from UI (and from postman) was keep on failing. 
The difference b/w the working (old account) & the non working version of my API was that - the value of Endpoint request body after transformations was having all headers and other params in the failing request , instead of the actual JSON request body

Sun Feb 14 18:31:38 UTC 2021 : Endpoint request body after transformations: {"resource":"/contactMeToCallSES","path":"/contactMeToCallSES","httpMethod":"POST","headers":null,"multiValueHeaders":null,"queryStringParameters":null,"multiValueQueryStringParameters":null,"pathParameters":null,"stageVariables":null,"requestContext":{"resourceId":"t90fph","resourcePath":"/contactMeToCallSES","httpMethod":"POST","extendedRequestId":"av3lqHDCSK4FtNA=","requestTime":"14/Feb/2021:18:31:38 +0000","path":"/contactMeToCallSES","accountId
...........
Sun Feb 14 18:31:39 UTC 2021 : Endpoint response body before transformations: null
Sun Feb 14 18:31:39 UTC 2021 : Execution failed due to configuration error: Malformed Lambda proxy response
Sun Feb 14 18:31:39 UTC 2021 : Method completed with status: 502

Proxy integrations cannot be configured to transform responses.

In the working version the value of Endpoint request body after transformations was as below --> Integration Request Type is LAMBDA

Sun Feb 14 18:57:09 UTC 2021 : Endpoint request body after transformations: {
    "firstname": "fname",
"lastname": "lasNm",
"phone": 1231231231,
"email": "email",
"desc": "testing from connect4wree"}
Sun Feb 14 18:57:09 UTC 2021 : Sending request to https://lambda.us-w

Root Cause & Solution: The Integration Type I selected was Lambda Proxy instead of Lambda



          How to enable CORS for LAMBDA_PROXY Integration Type?

                Set the CORS header in the response created from LAMBDA as below. API Gateway is                     just a pass  through in case of LAMBDA_PROXY



Usage Plans for API Gateway - Throttle the requests


 

 


Lambda (Glue b/w services, Serverless-means you don't manage the server)

Pay for processing time.

Chat History Pull

Order history App



 

 Transaction Rate Alarm



Uses

Can trigger based on time/fixed schedule - for periodic batch runs

Lambda Triggers - Many services can generate triggers to Lambda.


 

Language Support 

 


 

S3 Event Notification settings - for any event, with destination as Lambda, SQS or SNS topic


Lambda & Data Pipeline

Lambda & Redshift (DynamodDB can store the state info, & helps to batch data in to RS)


Lambda & Kinesis (Lambda will get a batch of stream records)
In Kinesis Data streams - partition key (same shard)
Kinesis Data Limitations
Produce using SDK, KPL or  Kinesis Agent

Java SDK - PutRecord (500 rec/call), GetRecord?  

 Consmer - SDK, KCL, FireHose, Lambda 

Kinesis Analytics - Serverless

Lambdas have a special behaviour when it comes to processing Kinesis event records. When the Lambda throws an error while processing a batch of records, it automatically retries the same batch of records. No further records from the specific shard are processed.

         https://www.youtube.com/watch?v=G9nSwSd64RU - Kinesis-Lambda (java8) 
Lambda code editor will not support Java 8 - so write and upload the JAR https://maven.apache.org/plugins/maven-assembly-plugin/usage.html 

                                      

Memory, Timeout, IAM Role


Code model






 


 













 

 

Sunday, January 17, 2021

Angular JS - notes

 

####### Introduction to Angular ######

##Angular (ng) High-level info (Angular 7)

Angular 2+ (‘Angular’) is a Typescript-based opensource front-end web application platform (originally from Google)

TypeScript (.ts) => String Typing, Decorators (Annotations)

--> ES6 (ECMAScript) => Classes, Modules, Arrow Functions

TypeScript Compiler generates Java Script (human-readable) --> Compile in to ES5 or ES6 

SPA - Single Page Application 

- AJAX & HTML5 - HTML Fragments (mini--views)

Clean separation of code that renders UI & the code that does the application Logic

Dependency Ingection - loose coupling b/w components and services

Automatic change detection (UI gets reflected automatically)

Comes with Rx.js - subscriptionbased processing of asynchronous data  - eliminated callback hell

Angular CLI - The scaffolding and deployment tool - spares developers from writing the boilerplate code and configuration scripts.

##Responsive Web Design (RWD)

- Web desgin approach - optimal viewing experiance on wide range of devices

- Responsive theme's layout adjusts to your devices screen size

##WorkSpace Setup


Node-v10.x https://nodejs.org/en/download/

Visual Studio Code

Angular CLI - npm install -g @angular/cli


##First Project (ng way)


#1 Open Terminal window --> go to your folder where you want to create the project

--> ng new <<your-project-name-here>>

#2 Launch the Angular application by running the command

--> ng serve --o

#3 Launches the index.html in a browser @ http://localhost:4200 (port might vary)

##Deployment and other info - https://angular.io/guide/deployment


##Build and Deploy as a JS

Ng serve

• In-memory compilation running on local server

• File changes trigger reload for iterative development

Ng build

• Creates output folder ‘dist’

• Consolidates app into a few js files

• Assets folder replicated

• Must change base href for deployment environment


##First Project (npm way)

Go to desired folder in Terminal and run  --> npm init -y

look at files, especially package.json

Add dependencies – Angular, SystemJS, Live-server, TypeScript compiler

Run the install command to install the above dependencies --> npm install

Copy the following files 

                    - SystemJS Config file, index.html, main.ts, app.module.ts and app.component.ts

Launch the Angular application by running the command

Launch the index.html in a browser @ http://localhost:4200/index.html (port might vary)

## Angular Project Structure 

#Angular.json


"outputPath" is the location where the ng build command places the condensed js files for the entire app

index.html is the single page in which the Angular App is injected

main.ts is the “main” Angular entry point

polyfills.ts contains browser compatibility polyfills and application polyfills.

styles.css is where global styles can be placed. Any CSS rules you place here are injected into the DOM in a <style></style> tag

"assets" array describes the location of static file assets

#Index.html

#main.ts --> calls bootstrap function

#app.module.ts --> being bootstrapped - to app component

#app.component.ts --> @Component decorator

#app.component.html --> 

################ Basic Concepts and Binding ##########


• TS / JS Primer
• Types
• Shapes
• Spread and Rest Operators
• Classes and Interfaces
• Decorators
• Arrow Functions

The Data Binding process in Angular 7
• Interpolation
• Property
• Event
• Two-Way
## Types (Strict Type checking)

function add (a:number, b:number){
  return a+b;
}
add(‘5’,6);//Compilation error
add(1, 3);//works
#Types: Boolean, number, string, arrays [], object literals{}, undefined, null, enum, any and void
    let hostname: string = "Sree";
let list: number[] = [1,2,3];
enum Color {Red, Blue, Green, Black};
let c: Color = Color.Black;
#Types inferred if no type is given
let a = 123;
## Spread Operators (spreading array into positional arguments)
var list = [1, 2, 3];
    list = [...list, 4, 5, 6];
    console.log(list); // [1,2,3,4,5,6]
## Destructiring 
var [x, y, ...remaining] = [1, 2, 3, 4];
console.log(x, y, remaining); // 1,2,[3,4]
## Rest Operators (like varargs in java) - accept multiple arguments in your function and get them as an array.
function fnTakeItAll(first, second, ...allOthers) {
console.log(allOthers);
}
fnTakeItAll('cat', 'mat'); // []
fnTakeItAll('cat', 'mat', 'bat', 'fat'); // ['bat','fat']
## var vs let

#var - Variables in JavaScript are function scoped
var foo = 123;
if (true) {
var foo = 456;
}
console.log(foo); // 456
#let - variables with true block scope
let foo = 123;
if (true) {
let foo = 456;
}
console.log(foo); // 123
 
## Union of types (can be al alternative to Inheritance)
export enum Color {RED = ‘red’, BLUE = ‘blue’, WHITE = ‘white’}
we could use union types and get similar benefits in a much shorter fashion:
export type Color = 'red' | 'white' | 'blue';
export type Optional<T> = T | undefined;
let user: Optional<User>;
export type AuthAction = LoginAction
                          | LoginSuccessfulAction
                          | LoginErrorAction
                          | LogoutAction
                          | LogoutSuccesfulAction;
type Age = number | string;
let numAge: Age = 50;
let strAge: ge = "Fifty";
function printAge(age: Age ): string{
   return "Is your age, ${age} ?"
}
## Classes & Interfaces
#Interfaces - are contracts - cant instantiate
interface Person {
firstName: string;
lastName: string;
}
# Classes - can be instantiated by TS
class Employee implements Person{
    firstName: string;
lastName: string;
}
## Shapes
Any two JavaScript objects (despite from different classes) are considered equivalent, 
if they are composed of same type of attributes
## Decorators
https://www.typescriptlang.org/docs/handbook/decorators.html
https://www.youtube.com/watch?v=3Rgv2UWK2Bo
Decorators are functions which will return functions, will/can have metadata
Auxiliary functions that can be hosted by Classes, Methods, Properties, Parameters, or Accessors
##Types of Decorators##
Class Decorators: @NgModule, @Component
Property decorators: @Input, @Output
Method decorators: @HostListener (event decorator)
Parameter decorators: @Inject
Any function can be used as a decorator
** Decorator - executed at the time of class evaluation (not instantiation) **
function myDecorator(prefix?: string){
return (constructor:any) => {
console.log(constructor);
console.log("decorator evaluated");
constructor.prototype.message = prefix + constructor.name;
}
}
@myDecorator("Hello ")
class World{
  message: string;
}
let w = new World();
console.log("Class Decorator ");
console.log(w.message);
console.log("-------------");
#Decorator Factory - We can write a decorator factory in the following fashion:

function color(value: string) {
  // this is the decorator factory
  return function (target) {
// this is the decorator
// do something with 'target' and 'value'...
  };
}
## Data Binding - Data binding signifies how and what kind of data is bound between a component and its template.

#1# Interpolation – binds component properties in output template. It uses {{}}.
Syntax : {{inrerpolation}}  means --> {{valueTo bind}}
<span>{{title}} App is running !!! </span>
#2# Property Binding – flows data from the component to the element. Uses []
<span [style.color]="componentStyle"> Some colored test!!</span>
#3# Event Binding – flows data from an element to the component. Uses ()
<button (click)="alertTheWorld()">Click Me</button>
#4# Two-Way Binding – is a combination of the Event and Property Bindings. 
Used along with the ngModel object. 
*Must Import FormsModule
          <input [(ngModel)]="dynamicValue" 
             placeholder="Watch the text update !" 
             type="text">
         <span>{{dynamicValue}}</spn>

## Component

Components form the building blocks of an Angular application.

To create a Component, issue the following command

ng g component <component name>

#Practice

1. Make Developer Class (ng g class Developer)

1. firstName: string

2. lastName: string

3. favoriteLanguage: string

4. yearStarted: number

2. Make new component called ‘bio’ (ng g component bio)

1. Import Developer Class

2. Create instance of a developer inside of the constructor

assign to a property called ‘dev’

3. Display dev component in bio.component.html

4. Add <app-bio></app-bio> to app.component.html

5. Create toggle switch using ngIF to only display bio component

if link is clicked

Use the "selector" value of the new component in the app.component.html to import it there

##  Component Lifecycle

• The lifecycle of a component is managed by Angular itself.

• It manages creation, rendering, binding data-bound properties etc. and also

offers the feature “hooks” that allows responding to key lifecycle events.

• Here is the complete lifecycle hook interface inventory:

• ngOnChanges - called when an input binding value changes.

• ngOnInit - called after the first ngOnChanges.

• ngDoCheck - called after every run of change detection.

• ngAfterContentInit - called after the component content is initialized.

• ngAfterContentChecked - called after every check of component content.

• ngAfterViewInit - called after the component's view(s) are initialized.

• ngAfterViewChecked - called after every check of a component's view(s).

• ngOnDestroy - called just before the component is destroyed.

##  Elvis Operator:

• If a property, that does not exist, is referenced in a template, an

exception is thrown.

• The “Elvis Operator” is a simple and easy way to guard against null and

undefined properties.

• It is denoted by a question mark immediately followed by a period “?.”.

<md-input-container>

<label>Type to see the value</label>

<input md-input type="text" />

</md-input-container>

<strong>{{input?.value}}</strong>

##  Structural Directives

•  ngIf adds and removes elements in the DOM based on the

results of an expression.

•  ngFor is a repeater directive that outputs a list of elements by

    iterating over an array.

<div *ngIf="emps">

<div *ngFor="let emp of emps">

<p><span class="bold">First Name:</span> {{emp.firstName}}</p>

<p><span class="bold">Last Name:</span> {{emp.lastName}}</p>

<p><span class="bold">Department:</span> {{emp.department}}</p>

<hr />

</div>

</div>

##  Introduction to Routes

• Routing allows to:

– Recover browser history functionality which is otherwise lost with SPA

– Maintain the state of the application.

– Implement modular applications.

– Implement the application based on the roles (certain roles have access to

certain URLs).

• Routes are injected into <router-outlet></router-outlet>. 

This is most commonly placed in app.component.html below any navigation bars or content

that you want to appear on every page


You can add routing after the fact

• Angular best practice is to create a separate, top-level module dedicated to routing

ng g module app-routing --flat --module=app


• - - flat flag will place the module in the top level

• - - module=app will add the import for app-routing inside of AppModule


## Route Configuration

Routes are configured, using the Routes type, which is an array of route objects. 

The route object is composed of the following attributes:

– Path: URL to be shown in the browser when application is on the specific

route.

– Component: Component to be rendered to, when the application is on the

specific route. This is the output of the router link.

##app-routing.module.ts## sample 

import { NgModule } from '@angular/core';

import { Routes, RouterModule } from '@angular/router';

import { HomeComponent } from './home/home.component';

import { EmpComponent } from './emp/emp.component';

const routes: Routes = [

{path:'', component: HomeComponent},

{path:'bio',component:EmpComponent}];


@NgModule({

  imports: [RouterModule.forRoot(routes)],

  exports: [RouterModule]

})

export class AppRoutingModule { }

## redirectTo ## Routes may redirect to other routes, using the redirectTo attribute.

const routes: Routes = [

{ path='', redirectTo= 'emp', pathMatch ='full' },

{ path='emp', component:EmpComponent }];

##Route Navigation##

• In the view template, routerLink directive may be used inside of

an anchor tag to add links that point to the defined routes.

<a routerLink:"/component-one" > Component One </a>

#Programatically do it this way

import {Router} from '@angular/router';

constructor (private router: Router){}


this.router.navigate("/emp")

##  BootStrap Stylesheet - industry standard style for Angular

<!-- google for bootstrap style sheet link to get below link -->

<!-- Add below link under head section of index.html -->

<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0/css/bootstrap.min.css"

integrity="sha384-Gn5384xqQ1aoWXA+058RXPxPg6fy4IWvTNh0E263XmFcJlSAwiGgFAW/dAiS6JXm" crossorigin="anonymous">


##  Services in Angular ##

• Create a Service using the command:

ng g service <<ServiceName>>

• Services expose methods (supposedly business logic) as

API along with optional public properties.

• Services are dependencies injected by Angular, which

maintains each service as a singleton.

• The @Injectable decorator is used to mark a Typescript

class as a Service.

• providedIn: Determines which injectors will provide the

injectable, by either associating it with an @NgModule or

other InjectorType, or by specifying that this injectable

should be provided in the 'root' injector, which will be the

application-level injector in most apps


## Injectable Decorator

## Dependency Injection

• Import the service class

• Define it within the constructor parameters

• It should have the ‘private’ access modifier

## Route Parameters


Parameters are sent using :paramName embedded on to the router paths:

const routes: Routes = [

{ path='', redirectTo= 'product-list', pathMatch ='full' },

{ path='product-details/:id', component: ProductDetails }];

The parameter can then be sent using routerLink and interpolation:

<a routerLink="product-details/{{product.id}}"> {{product.name}} </a>

OR the routerLink directive may be used and supplied with parameters 

to be automatically sent, while invoking a router link.

<a *ngFor="let product in products"

[routerLink]="['/product-details', product.id]">

{{product.name}}

</a>


## Reading Routing Parameters using ActivatedRoute ##

– Angular provides the ActivatedRoute service, which in turn supplies 

a paramMap property which contains the parameters.

import {ActivatedRoute} from '@angular/router';

@Component.....

export class TestComponent implements onInit{

id;

constructor(private route: ActivatedRoute){}

ngOnInit(){

this.id = this.route.snapshot.paramMap.get('id')

}

}

## Navigate to parameterized routes via code

Navigating to parameterized routes internally:

– Create a Router object and use the navigate method.

– Pass an array containing the main route and the parameter

(just like routerLink directive on previous page) 

@Component ....

export class TestComponent{

constructor(private router: Router){}

goToProductDetails(id){

this.router.navigate('/product-detail', id);

}

}



## Creating in app-routing.module.ts

## Linking

## Retrieving parameters in component class

## Child Routes

================================================