今日のつぶやき

Twilog

[Twitter]今日のつぶやき ff.im/bHzXP #

会場到着。寝坊して朝食とれなかった。 #

#pdc09 『Building Scalable and Relizalbe Applications with Window Azure』 #

#pdc09 スライドがでなくなった。 #

#pdc09 windows error recoveryが走り出した。 #

#pdc09 復活 #

#pdc09 high availability, scalability, lifecycle management #

#pdc09 data scalability, scaling computation and workflow, lifecycle management - upgrades and versioning
,- agenda #

#pdc09 volatifle storage, persistent storage
local storage, caches (e.g., appfabric cache and memcached) #

#pdc09 blobs, tables, queues #

capacity 100TB, throughput - up to a few handred megabytes per socond, Transactions - up to a few thousand requests per socnd #

#pdc09 for hgher through put blob content use Windows azure content delivery network #

#pdc09 we group your blobs, entities, and messages into partitions #

#pdc09 automatic, load balancing #

#pdc09 Master systemがreassign partitionsする #

#pdc09 time beetween offload to reload is on the order of seconds #

#pdc09 time to decide to load balance is on order of minutes #

#pdc09 partitioning of Data Objects

• allows the system to automatically scale out access to your data to meet its peak traffic demands #

  #dpc09 all objects with the same partition key value are always grouped into the same partion #

  #pdc09 partition key used blobs blobname, messages queue name , entities - application defined partition key for table #

  #pdc09 throughput

single queue and single table partitoin
single Blob #

#pdc09 Latency
typicallly around 100ms (for small trans) #

#pdc09 but once in awhile can increase to a few seconds curing heavy spikes while load balancing kicks in #

#pdc09 compute service model

• the topology of your service,

types of roles, how the roles are connected #

#pdc09 web role, worker role #

#pdc09 - configuration of the service #

#pdc09 Web + Worker Role Service Model #

#pdc09 due to application failures, upgaredes or hardware failures
Use multiplie instances of each role type so the service * #

#pdc09 feed the work items to the worker roles #

#pdc09 break job into work items (optiona "map" step) #

#pdc09 aggregagte ("reduceW)" #

#pdc09 loosely coupled work with queues #

#pdc09 worker-queue workflow #

#pdc09 Queue workflow concepts #

#pdc09 guaranttee delivery (two-step consumption) #

#pdc09 a message may be processed more than once. #

#pdc09 assume messages put into same queue can be processed in any order (best effort FIFO) #

#pdc09 make work items idempotent - work is repeatable and can be done multiple times #

#pdc09 Job Manager
generating the load
distributing the load
monitor progress
manage resources
aggregate results #

#pdc09 riskmetrics case study
focused on finacial risk management
they use th ejob manager workflow model #

#pdc09 currently feed the work items to 2,000 worker roles
planto run 10,000+ worker roles #

#pdc09 scaling queue throughput
batch work items into Blobs #

#pdc09 group together many work items int a blob #

#pdc09 use multiple queues

• job manger
• workers #

  #pdc09 continuation for long running work items

• want to continue on failover #

  #pdc09 high level approach

break work item into smaller and repeatable steps. #

#pdc09
record progress after each step indexed by work item #

#pdc09 In-Place Rolling Upgares
specify number of upgrade domains in service model

• break your roles evenly over the number of domains #

  #pdc09 rolling upgrade

• work each upgrade domain one at a time
• upgrade just he roles in the current domain #

  #pdc09 benefits

minimizes availability loss
allows local state to be used across upgraddes
catches application upgrade issues early. #

#pdc09 versioning with rolling upgrades
always assume you wil have old and new running side by side in your serivce

• verison everything #

  #pdc09 protocol change with rollng upgrade

all nodes stil initiate contact sending old protocol version #

#pc09 version up時に新旧プロトコルが並存する時の処理方法。Rolling upgrade。 #

#pdc09 tabel schema change
have a version property in each entity
types of schema change #

#pdc09
Adding non-key properties

• use ignoremissingproperties

removeing non-key properties #

#pdc09

• use ignoremissingproperties and replaceonupdate

change partition key and row key
copy new table.(?) #

#pdc09 upgrade domainごとにプロトコルやスキーマのバージョン管理ができると運用がずいぶん楽になる。 #

#pdc09 V2 client
updated all entities to _V2 on first use and start putting real values in new property #

#pdc09 v1.5 client
if entity version is v1
store the default value in the new property, and don't change version #

#pdc09
if entity version is v2
use the new valuwe an update it. #

#pdc09 table schema rolling upgarde
v1 code - only understands version 1 #

#pdc09

• v1.5 code creates entities in version 1 #

  #pdc09

• processes an existing entity based on its curren tversion 1 or 2, and doesn't convert any entity #

  #pdc09 data scalability

leverage partition ning #

#pdc09 scaling computation
loosely coupled workflow with queues #

#pdc09 upgrade and versioning

• with in-place rolling upgrades, always assume aold* #

  #pdc09 『Scaling out Web Applications with Microsoft SQL Azure Database』 #

  #pdc09 SQL Azure = Highly scaled out relational database as a service #

  #pdc09 customer value propes

self-provisioning and capacity on demand
symmetry w/ on-premises database platform #

#pdc09
automatic high-availability and fault-tolerance
automated DB maintenance (infrastructure)
Simple, flexible pricing - pay as you grow #

#pdc09 T-SQL (TDS)で通信 (from in-premise, cloud both) #

#pdc09 workload, cpi/IO, storage #

#pdc09 partitioning - classic technique #

#pdc09 classic scale-out
highly available service on top of commodity hardware* #

#pdc09 pre-conferenceの人がゲストででてきた。 #

#pdc09 パーティショニング大事 #

#pdc09 transactional requirementsとstorage requirementsによる4象限でアプリケーションの戦略を考える #

#pdc09 low/low
single database, no partitiooning #

#pdc09 storage hight/trans low
partitioned data
partitioninng based on application requirements(storage) #

#pdc09 high/hight
partioned data
partitioning based on application requirements(IOPS, Storage or both) #

#pdc09 trans high/storage low

#

#pdc09 scale out patterns

range - break range into chunks

• good for range based queries #

  #pdc09

hashing - apply hash to partitioning keys

• good for distributing values #

  #pdc09

can suffer from hotspots depending on workload
need to accurately estimate workload requirements #

#pdc09 product
stock table can become hot
range partitioned
certain reference data replicated to customer order databases #

#dpc09 customer order
need to evenly distribute load
hash partitioned
start with 1gb database #

#pdc09
move to 10 gb databases should addional capacity required #

#pdc09 application partitioning
schema design & management #

#pdc09 schema design should avoid cross database joins and transactions #

#pdc09 DDL should be upgrade resilient and idempotent #

#pdc09 application code should either
support multiple schemas during an upgrade
sotop processing requests during an upgrade #

#pdc09 Request Routing
application needs to be "partition" aware

• partion is a unit of transactinal consistency #

  #pdc09

• multiple partions are idependent of each other #

  #pdc09 reference data & syncrhonization

• avoid cross database joins
• replicate refrnce data for performance resons
• #

  #pdc09

• option 1 Sync Frameork
  • Works Greagt!!!
  • Host in Azure Worker Role #

    #pdc09

Option 2 -Manual Update Scripts #

Fan Out Queries
issue query to maultiple databases in parallel and aggregate the result #

#pdc09
muse multiple connections and multithreading for increaswed performance #

#pdc09 Coming scale-out Support
Better tooling for developers and administrators: #

#pdc09

• dynamic database partition splits
• ability to merge database partitions
• improved schema management across database partitions #

  #pdc09 Database features for scale-out application patterns

connection management
fan-out query support #

#pdc09 Scalling out provides virtually unlimited storage and better performance
range and hash scale-out patterns #

#pdc09 consitertion when scaling tou
schema desing ana management
request routing #

#pdc09

• reference data
• fan out queries

future scale-out support #

#pdc09 SQL AzureのScale Outはpartitioningにつきるということかな。
分割したデータベース間の同期もissue。 #

#pdc09 『Workflows Services and Windows Server AppFabric" (formerly codename "Dublin")』 #

#pdc09 今回のPDCはクラウド的にはAppFabricとSQL Azureがキーだと感じている。 #

#pdc09 work is increasingly distributed
this leads to complex code for cooridnation #

#pdc09
configuring an environment to deploy and run serivices is difficult #

#pdc09
minimal management and visibility into the operation of services in production #

#pdc09 persistence behavior, tracking behavior #

#pdc09 message, activity library (receive, send) #

#pdc09 management endpoint #

#pdc09 evolution of WAS/IIS and Windows Application Server role to run and manage WCF and WF services #

#pdc09 services, workflows #

#pdc09 persistence, hosting, monitoring, caching #

#pdc09 下に.NET Frameworkがある。その下にRuntime Databases, IIS/WAS、Windows Server。 #

#pdc09 AppFabric Services Workflows + Visual Studio 2010のデモ中。 #

#pdc09 IService (Basic Http Binding_IService) #

#pdc09 Webサービスベースなのかな #

#pdc09 monitooring, workflow persistence, availability, performance, security #

#pdc09 serviceの糊言語という視点でworkflowを考えるとよい。糊言語の実行系でavailability、monitoring, securityをハンドルしてくれる。 #

#pdc09 糊言語として考えた場合、serviceのcontractをどのように宣言するかが問題となるけど、そのあたりはどうなってるのかな。 #

#pdc09 coordination of work
asynchronous messaging
coordinate multiple service calls #

#pdc09 Message Correlation
the mechanism for relating messages to each other or to application instance state #

#pdc09 context based correlation - New in .NET 4 #

#pdc09 keyとentity idのマッチングを管理する #

#pdc09 デモ中。リアルシステムをグラフィカル言語で編集するのは大変そう。 #

#pdc09 Monitoring
consistent WCF and WF instrumentation integrated with Event Tracing for Windows (ETW) #

#pdc09
Improved performance means less impact on your appliations #

#pdc09
visibility into the operation of services #

#pdc09
WCF trace events
WF tracking events #

#pdc09 Event Collector Service
A windows service installed iwth the Windows Server AppFabric #

#pdc09
Collects WCF and WF instrumentation events into the Monitoring DB #

#pdc09 モデリング的には業務ユースケースやシステム・ユースケースをどのようにworkflowに落とし込むのかという点が切り口。 #

#pdc09 activity diagramはパスしてよいだろう。workflowで直接記述する。 #

#pdc09 ユースケースからサービスとサービスの実行によって引き起こされるエンティティの状態遷移をどのように記述するか。状態遷移の集合をワークフロー全体の状態遷移として集約できるか。モデリングとの連携ではこのあたりが効いてくる。 #

#pdc09 Persistence
Storing WF Application State
Ensures that durable workflows can be recovered and processing resumed #

#pdc09 Scenarios
reliability - long running workflows
availability - recovery when application or machine crashes #

#pdc09
scalability - a workflow unloads idle instances from memory; the retry loading of locked instaces #

#pdc09 ジョブ管理だね。 #

#pdc09 running - suspended - completed #

#pdc09 こういうところの技術は汎用機の時代からあまり変わっていない #

#pdc09 ジョブ管理とかTPモニタとかをクラウド上でどのように実現していくのか。いずれ、論点として浮上する。AppFabricでAzureまで展開済み、これからのどちらだろう。 #

#pdc09 Instance Availability #

#pdc09 AppFabricをジョブ管理と考えるとWorkflowはJCLと考えるとよい。 #

#pdc09 Summary
workflow is ideally suited for business services
workflow serivces handle complex coordination #

#pdc09
the windows server appfabrick suports wcf and workflow services by
provideing runtime databases #

- dcapabilitites for monitoring, persistence, hosting and caching

• tooling and scriptingt to manage and monitor wcf and wf service #

  午前おわりー #

  #pdc09 『The Future of Database Development with SQL Azure』 #

  #pdc09 Project Houstonの話らしい。 #

  #pdc09 lifecycle

requirements analysis
schema design
implementation
maintenance #

#pdc09 Newly added suport in SSMS R2 is changing the game #

#pdc09 SQL Server 2008 R2デモ。
古い手順のデモかな #

#pdc09 古いのでは生成したscriptで何かするという手順 #

#pdc09 challenges
no uniform DB project system and workflow #

#pdc09
no ability to capure deployment intent
lack of T-SQL designers for breadth developer #

#pdc09 development, deployment, management #

#pdc09 development
Data-tier application project
New T-SQL IDE in VS 2010 #

#pdc09 deployment
easier to deploy and upgrade Data-tier Apps from vS and SSMS #

#pdc09 management
Manage @ Scale: SQL Server Control Point #

#pdc09 concepts
Data-tier application Project

• first class T-SQL/DAC development experience in Visual Studio
• biuld output = .dacpac #

  #pdc09 Data-tier Application Component

• improves collaboration between developer and DBA #

  #pdc09

• Moves developmers from a procedural model to a declarative model #

  #pdc09

• contains the desired shape of the appliatoin #

  #pdc09 data-tier application component

• schema
  • logical, physical #

    #pdc09

• DAC deployment profile
  • deployment requirements, management policies, failover policies #

    #pdc09 managing multiple database environments

dev, test, prod #

#pdc09 phasee transitions are ypically a manual process guided by a devlopment methodology #

#pdc09
manual processes are error prone #

#pdc09 managing multiple database environments
Dev DB, Test DB, UAT DB, Prod DB #

#pdc09 combine existing technologies together

• business rules
• MSTest
• DACPACs #

  #pdc09 ensure that all updates are fully tested

Prevent any fat-fingering
Allows youto ** #

#pdc09 "Houston" #

#pdc09 デモ中 #

#pdc09 DB間の移行を自動的に行うツール? #

#pdc09 Project Houston will provide a easy to use cloud based database development experience #

#pdc09 you can develop and interact w/ SQL Azure databases in that same way an d leveraging the same skill set you are have for SQL Server. #

#pdc09 『Using the Microsoft Sync Framework to Connect Apps to the Cloud』 #

最後のセッション #

#pdc09 Synchronization
incremental changes
unreliable and/or slow network
confiict detection and resolution
topology flexibility #

#pdc09 利用者
windows 7 PMM Data, SyncToy, SharePoint 2010, SharePint Workspace 2010, Windows Live Favorites, Credit Suisse, #

#pdc09 PricewaterhouseCoopersもユーザなのかぁ。 #

#pdc09 sync application, sync orchestrator, sync provider, sync runtime, store
<- SyncFX Application Components #

#pdc09 synchronizaing data between SQL Server and SQL Azure #

#pdc09 SQL Azure Data Syncのデモ中 #

#pdc09 SQL Azure DatabaseとSql Server Local Database間でテーブル単位で同期を指定している。 #

#pdc09 SqlAzureSyncProvider, SyncOrchestrator, SyncStagedProgressEventArgs, DbSync*** #

#pdc09 orchestrate.Syncrhonize() #

#pdc09 Offline-Capable Applications
availability when occasionally connected or disconnected #

#pdc09
interactive clients query lots of data
interfactive clients require minimal query latency #

#pdc09
reduce server-side load
reduce network usage #

#pdc09 examples:
sales, deliveries, real-estate agents, auditors
branch office, retail stores
Outlook Exchange Cached Mode #

#pdc09 SQL Compact - SQL Compact Sync Provider
SQL Azure Provider - TDS - SQL Azure
Sync Orchestratorで同期する #

#pdc09 Offline-Capable ApplicationはSQL Compactにデータを書き出して、orchestratorに同期を指示する #

#pdc09 SqlAzureSyyncClient_AdventureWorksLT2008_Offline.Synchronize(); #

#pdc09 デモが遅い。無線LANが重いせいだと思われる。 #

#pdc09 SQL Azure Proxy Provider - WCF Sync Endpoint - SQL Azure Provider (SyncFX V2 Three-Tierの場合) #

#pdc09 clients authenticate with windows azure application *** #

#pdc09 looking ahead to syncFX V3
Windwos azure上のsync endpointに対してwindows, silverlight, HTML, Other Platformsが同期する。 #

#pdc09 define an HTTP-based sync protocol
Ohter Platformsとの同期に使用する。 #

#pdc09 sync smarts on server, not on client #

#pdc09 minimal client & store requirements - HTML #

#pdc09 .NETFX - 専用プロトコル? #

#pdc09 相当強力なモバイルソリューション。 #

#pdc09 HTML5との連携も視野に入っている。 #

#pdc09 HTML5では、ローカルDBとサーバ上のDBのそれぞれにアクセスするAPIがあるけど、前者はSQL Compact+SyncFX V3、後者はTDS経由でSQL Azure、REST経由でWindows Azure Storageといった連携ができそう。 #

#pdc09 data sync service
Service
end-user application
little or no development required, mainly configuration #

#pdc09
provoide more out-of-the-box, but narrower scop
pay for use #

#pdc09 capabilities
distributed data manaement, client management & monitoring, user authentiatkon & authorization #

#pdc09 Nov'09 SyncFX Poer Pack for SQL Azure CTP
H1'10 - SyncFX V2.1
H2'10 - SyncFX V3 #

#pdc09 v2.1 full support for windows azure platform
sql azure provider and syncfx deployable to a web role #

おわりー #

会場の無線LANが動かなくなったので、ホテルから送信。 #

Automatically shipped by LoudTwitter